A path aggregation network with deformable convolution for visual object detection

The structure of single-stage detector

Although various single-stage object detection algorithms have been proposed, al-most all of them can be abstracted as three parts: backbone, neck and detection head (Wang, Bochkovskiy & Liao, 2023; Jiang et al., 2022; Liu et al., 2016; Zhang et al., 2018). The structure of the detection process is shown in Fig. 1, where the backbone is used to extract the features of different scales in the image, the neck conducts the fusion operation on multi-scale features extracted by the backbone, and the head output the predictions to the bounding box and the category of the object in the image.

In the two-stage object detection method, the region proposal network can generate a large number of redundant candidate boxes and thus consume a lot of computational time. In contrast, the single-stage detection algorithm abandons the region proposal stage and implements end-to-end object detection with YOLO series being the representatives. YOLOv1 (Redmon & Divvala, 2016) uses CNN to extract the features of the original image and directly performs bounding box regression and object classification. Compared to the two-stage method, it has a great improvement on computational efficiency, but its detection accuracy has no significant enhancement. In YOLOv2 (Liu et al., 2018), an anchor frame mechanism is introduced and the prediction of the location is no longer performed by blind and brute regression but by the computation of the offset of the pre-given anchor frame to the actual bounding box, which results in great improvement on detection speed and accuracy. YOLOv3 (Redmon & Farhadi, 2018) further adds FPN (Lin et al., 2017) as the neck to fuse multi -scale features for prediction, with the speed and accuracy of the model further enhanced. YOLOv3 once became one of the most widely used models in the industry.

Path aggregation networks

The current mainstream design of necks is to combine the feature maps of different scales. The fusion methods can be the addition of the corresponding elements of the features, such as FPN (Lin et al., 2017), SPP (He et al., 2015), etc., and also can be stitching of channel dimensions, including the path aggregation network structure used in YOLOv4 (Liu et al., 2018; Bochkovskiy, Wang & Liao, 2004). As high-level features often include high-level semantic information of the image, the corresponding feature maps are generally small and thus upper sampling must be performed before they are fuse them with low-level features. On the contrary, low-level features generally contain low-level information of object shapes but have large feature maps, so down sampling must be performed before they are fused with high-level features. Figure 2 illustrates an example of PAN structure. In this example, the inputs of the PAN are the features of four different scales extracted from the backbone on the leftmost side. Here, {P₂, P₃, P₄, P₅,} denotes the feature levels generated by upper sampling and feature fusion. Specifically, features of level P₄ are obtained by summation of the input feature map of the same scale with the resultant features upper sampled from features of level P₅. By adding the features upper sampled from features of P₄ to the original feature map of the same scale, the features of P₃ can be obtained. Similarly, the features of P₂ are generated by adding the features upper sampled from the features of P₃ to the original feature map of the same scale. In this way, the top-down feature fusion is implemented. In the framework, {N₂, N₃, N₄, N₅} represents the feature levels generated by down sampling and feature fusion. Put it in detail, the features of N₂ are the same as those of P₂, the features of N₃ are resulted from the element-wise addition of the features of P₃ to the features down sampled from those of N₂, the features of N₄ are obtained from fusion of those of P₄ with the features down sampled from those of N₃, and the features of N₅ are generated from fusion of those of P₅ with those down sampled from the features of N₄. This is the bottom-up feature fusion in the PAN. Thus, the PAN outputs the features maps of four different spatial size (i.e., N₂, N₃, N_4, and N₅) which are then input to the head for prediction of the bounding box and object category.

Deformable convolutions

The sampling mode of the traditional convolution is fixed, and thus its receptive field cannot capture sufficient feature information. The method of increasing the receptive field is to stack convolution layers, but too deep convolutional layers often cause the loss of the low-level features and a large number of redundant parameters. The deformable convolution can change the convolutional sampling locations by a convolutional offset, which is learned during the process of network training (Dai et al., 2017). With adaptive adjustment of the sampling location, the deformable convolution can achieve the effect of increasing the receptive field without increasing the number of layers. The structure of deformable convolution is illustrated in Fig. 3. In the figure, the yellow part is the offset field (or offset map) trained during the network training process. The number of channels of the offset field is 2N, where N is the number of sampling points by the convolution kernel. By taking 3 × 3 convolution kernel as an example, the number of sample points is thus 9, that is, N = 9. When deformable convolution is performed, a total of 2N = 18 channel elements are taken from the same locations in the offset field as the current output locations, and they represent the offsets of nine sampling points in the x and y directions, respectively. By the vector addition of the offsets of each point in the x and y directions, the offsets of nine sampling points can be obtained separately.

For the input feature map X, the sampling point is first obtained by convolution based on a regular grid R, and the output feature map can be generated by weighted summation of the sampling points with the weights W. The regular grid R of the 3 × 3 convolution kernel is given by

(1) $$R=\left\{\left(-1,-1\right),\left(-1,0\right),\cdots ,\left(0,1\right),\left(1,1\right)\right\}$$which represents the locations of the central point and sampling points in the other eight directions. For point P₀ in the input feature map, the output of convolution is yielded by

(2) $$y\left(P_0\right)=\sum _{P_{n\in R}}w\left(P_n\right)\cdot x\left({\overline{P}}_0+P_n\right).$$

Let $P_0$ denote the center point in the input feature map, { $P_1$, $P_2$,…, $P_n$} represent the eight sampling points in the surrounding directions, $w\left(P_n\right)$ indicate the weight associated with the sampling point $P_n$, and $x\left(P_0+P_n\right)$ denote the input feature value at the position offset by $P_n$ from the center $P_0$. The convolutional output is computed by performing a weighted sum of the feature values from these sampling points around $P_0$. This mechanism enables the extraction of local features from the input feature map and facilitates feature transformation through spatially learnable weight adjustments.

In the deformable convolution, the regular grid R is augmented with offsets $\{\mathrm{\Delta }{P}_n|n=1,2,\cdots N\}$, where $N=\left|R\right|$. Thus, the output feature map y by deformable convolution is

(3) $$y\left(P_0\right)=\sum _{P_{n\in R}}w\left(P_n\right)\cdot x\left(P_0+P_n+\mathrm{\Delta }{P}_n\right)$$

Since the offsets are generally fractional, the pixel value of the sampling point x should be obtained via bilinear interpolation, that is, it is determined by the pixel values of the surrounding points, so that the receptive fields can be increased.

The proposed method

In this work, we propose a framework of neck combining deformable convolution and path aggregation network (DePAN) for the single-stage object detection. The structure of DePAN neck is illustrated in Fig. 4. It takes as input the features of different size extracted by the backbone network. For the features of deep layers, DePAN neck first use a convolutional layer to extract features and adjust the feature dimension, and then undertakes upper sampling through conversion convolution; Next DePAN stitches the up-per sampled features with adjacent low-level features in the channel dimension, and then fuses the features through the DeConVBLOCK. This process corresponds to the fusion from top to bottom on the left part of the DePAN neck in Fig. 4. For the features of shallow layers, DePAN neck carries out down sampling and stitches the down sampled features with adjacent high-level features in the channel dimension, and then fuses the stitched features through DeConVBLOCK, and finally outputs the fused features to the head for prediction. This corresponds to the bottom-up fusion on the right part of the DePAN neck in Fig. 4.

Specifically, with the given input feature maps, for example X1 and X2 and X3, the proposed DePAN neck first obtains the features of the upper layer P3 by letting P3 = X3, and get the features of lower layers P2 and P1 by

(4) $$P_2=g\left(h\left(f\left(P_3\right),x_2\right)\right),P_1=g\left(h\left(f\left(P_2\right),x_1\right)\right)$$

where f(∙) is to perform convolution and conversion convolution on the input in turn, h(∙,∙) is to stitch the feature maps of the same size in the channel dimension, and g(∙) represents the operation of DeConvBlock on the stitched feature maps.

$P_1$, $P_2$, $P_3$ represent features at distinct hierarchical levels. $P_3$ is the upper-level feature, while $P_2$ and $P_1$ are lower-level features derived through corresponding function computations. For the input feature maps, $x_3$ is used to directly obtain the upper-level feature $P_3$ whereas $x_2$ and $x_3$ are utilized to compute the lower-level features $P_2$ and $P_1$ via function computations.

With the obtained feature maps of P3, P2 and P1, the DePAN neck further gets the features of the largest size on the output layer, N1, by letting N1 = P1, and get the remaining features N2 and N3 by

(5) $$N_2=g\left(h\left(S\left(N_1\right),P_2\right)\right),N_3=g\left(h\left(S\left(N_2\right),P_1\right)\right)$$where S(∙) denotes the convolution operation on the input feature map.

In the DePAN neck, DeConvBlock is composed of N stacked DeConv Cell components, whose structure is illustrated in Fig. 5. The branch structure is adopted in the design of the DeConv Cell. The input features go through the branch of the 1 × 1 convolutional layer, the branch of the 3 × 3 deformation convolutional and batch normalization, and the branch of a separate batch normalization. The results of the three branches are added up and the summation is then input to the activation function.

In the DEPAN NECK, stitching operation is carried out for feature fusion, and in DECONV cell, addition operation is directly performed to fuse the features. Table 1 is the details of parameter configuration in the DePan NECK applied to YOLOv6 (Li et al., 2022).

Each DeConvCell contains a convolution operation and a deformable convolution operation. Table 2 lists specific parameter configuration of the DeConvCell with four stacked DeConvBlocks. According to the parameter configuration, the shape information of the feature maps in each layer can be easily tracked.

The dimensions need to be adjusted in DeConvBlock1 and DeConvBlock 2, and the output dimensions of P2 and P1 are 256 and 128, respectively. In each DeConvBlock with 12 layers of stacked DeConvCells, the first layer of DeConvCell is responsible for adjusting the dimension with the output dimension of convolution and deformable convolution being the dimension of the expected output feature map; the convolution and de-formable convolution in the remaining 11 layers of DeConvCell do not change the dimension. The 12 layers of DeConvCell in both DeConvBlock3 and DeConvBlock 4 do not change the channel dimensions. The dimensions of the final feature maps of the three different scales output by the DePAN neck, i.e., N1, N2, and N3, are 128, 256 and 512, respectively. These feature maps are then input to the detection head. For different object detection network models, DePAN neck can be plugged and played and is convenient to add to the models, only needing to adjust the channel dimension slightly according to the actual situation.

Experimental settings

To evaluate the effectiveness of the proposed DePAN neck in object detection, we applied the DePAN neck in the YOLOv6 models of two volumes and tested on the CO-CO2017 (Lin et al., 2014) and PASCAL VOC2012 (Everingham et al., 2009) datasets, with the backbone, the head and the loss function the same as those of the original YOLOv6.

COCO2017 is a large dataset provided by Microsoft and contains a large number of labelled data for object detection, segmentation, posture estimation, behavior recognition and other tasks. The labeling information is stored in jason format. The dataset totally contains 118,278 images for training, 5,000 images for verification, and 40,670 images for testing. There are a total of 80 categories in the dataset and each image includes 3.5 objects on average. It is the most widely used dataset in the field of computer vision.

PASCAL VOC2012 is also one of the datasets commonly used in the field of computer vision. It contains 20 object categories, including the labeling information for various visual tasks such as target detection, segmentation, behavior recognition, and the label format is XML. For object detection tasks, PASCAL VOC2012 provides 5,717 images for training and 5,823 ones for verification; for segmentation tasks, the dataset offers 1,464 images for training and 1,449 ones for verification, along with the segmentation results.

All the experiments were run on a workstation containing four Tesla T4 graphics cards. The optimizer used in the model training was the stochastic gradient descent (SGD) algorithm. The size of the input image and the data augmentation was the same as those in the original YOLOv6. Batch size was set to 128 in all the experiments. Since the batch size is 256 in the original YOLOv6, the learning rate was reduced to 1/2 of the original setting, namely 0.05. The cosine scheduler was employed for the learning rate. Each iteration executed for 300 epochs and no pre-training weights were used. The experiments were implemented based on the PyTorch deep learning framework.

Experimental results for different models

We first used our proposed feature fusion framework, the DePAN neck, to the two YOLOv6 models of different parameter volumes, i.e., YOLOv6-N and YOLOv6-T to verify its effectiveness. Performance comparison among different YOLO models were made and the results are listed in Table 3, where the results in bold indicate the higher one between the accuracies of the improved model and the original model.

It can be seen from Table 3 that the detection accuracies by the baseline models, Yolov6-N and YOLOv6-T, on COCO2017 dataset are 35.9% and 40.3% of mAP^0.5:0.95, and 51.2%, and 56.6% of mAP^0.5. Yolov6-N* and YOLOV6-T* denote the YOLOv6-N and YOLOv6-T, whose necks are replaced by the proposed DePAN necks respectively. Thanks to the enhancement of feature fusion capabilities by the proposed methods, the model performance was significantly improved. Specifically, for the YOLOv6-N*, its mAP^0.5:0.95 increased by 0.7% from 35.9% to 36.5%, and its mAP^0.5 increased by 1.2% from 51.2% to 52.4%. For YOLOv6-T*, its mAP^0.5:0.95 increased by 0.8% from 40.3% to 41.1%, and its mAP^0.5 increased by 1.4% from 56.6% to 58.0%. Finally, a comparison was also made with the latest version of the YOLO series. There is not a big gap in mAP accuracy. And because the detector of the DETR series has a large number of parameters, although the operation speed of our detector is less, the effect comparison is not very ideal.

In addition, in order to verify the effectiveness of the proposed method in the detection of small objects, we also tested YOLOv6-N* and YOLOv6-T*, which adopts our proposed DePAN necks, on data of small objects in COCO2017. The results for mAP of the models are shown in Table 4. It is shown that compared to the baseline models YOLov6-N and YOLOV6-T, YOLOv6-N* and YOLOv6-T*, with the help of our DePAN neck, obtained higher accuracies in small object detection. Specifically, their mAP^0.5:0.95 increase by 0.5% from 16.5 to 17.0 and by 1.3% from 20.3% to 21.6%, respectively.

Ablation studies

In order to verify the effectiveness and generalizability of our proposed DePAN neck, we undertook the ablation studies on the number of stacked DeConvCells in the DeConvBlock with another dataset, PASCAL VOC2012.

We tested YOLOv6-N* and YOLOv6-T* and their baseline models on PASCAL VOC2012 dataset, training the models on the training set and verifying them on the verification set. The results are reported in Table 5. As can be seen, Yolov6-N*and YOLOV6-T* both had remarkable improvement on detection accuracies over their corresponding baseline models respectively. For YOLOv6-N*, it had an increase of 1.3% in mAP^0.5:0.95 from 47.1% to 48.4% and an increase of 0.9% in mAP^0.5 from 66.7% to 67.6%. For YOLOv6-T*, it had an increase of 1.1% in mAP^0.5:0.95 from 49.1% to 50.2% and an increase of 0.9% from 49.1% to 50.2%, and an increase of 2.3% in mAP^0.5 from 67.3% to 69.6%. We integrated the module on YOLO11-N. Compared with the detectors after the sixth generation, the mAP^0.5:0.95 increased from 40.9% to 46.5%, with a significant improvement in effect. From these results, we can see that the proposed method can also effectively improve the model performance in object detection on different datasets, indicating its good generalizability.

The DePAN neck proposed in this work has DeConvBlocks, each of which consists of repeatedly stacked DeConv cells. In order to investigate how the number of stacked DeConv cells influences the performance of the DePAN neck, we performed the experiments on PASCAL VOC2012 with four different settings for the number of cells. The results are shown in Table 6. When the numbers of the stacked cells in the four DeConvBlocks were 12, 12, 12 and 12, respectively, YOLOv6-N*obtained a mAP^0.5:0.95 of 48.4% and YOLOv6-T* 50.2% respectively. The results from this setting of the numbers of stacked cells are better than the results with the numbers set to 6, 6, 12, and 12 and the results with the number set to 12, 12, 6, and 6. It is shown that when the numbers of the stacked cells were set to 16, 16, 16 and 16 respectively, both of the models showed only slightly better performance than the model performance with 12, 12, 12 and 12 stacked cells in the four DeConvBlocks respectively. Therefore, in the other experiments in this work, we set all the numbers of stacked DeConv cells in the four DeConvBlocks to 12. Additionally, we have also conducted a similar experiment and applied it to YOLOv8 and YOLOv9. The relevant experimental comparison results are presented in Table 6. It can be observed that with the addition of DeConv cells, the experimental effect continuously improves, proving that our module is effective.

Experimental results on medical image dataset

In order to further demonstrate the applicability of our proposed method in re-al-word problems, we also performed experiments on medical imaging dataset. The dataset contains 1,076 CT images of breast nodules, including 476 benign cases and 600 malignant ones. The size of each image is 224 * 224. The tumor nodules were marked with DarkLabel labeling software. Benign and malign images together were divided into training set and verification set according to the ratio of 7 to 3. That is, there were 753 images in the training set, including 333 benign images and 420 malignant images, and 333 images in the verification set, including 143 benign images and 180 malignant images.

The experimental results are shown in Table 7, where Yolov6-N*and YOLOV6-T* are the models with the proposed DePAN neck added. It can be seen from the experimental results that on the breast CT image dataset, the mAP^0.5:0.95 obtained by YOLOv6-N* is 43.3%, higher than that of its baseline (41.9%) by 1.4%; the mAP^0.5 obtained by YOLOv6-N* is higher than that of its baseline by 4.2%. For YOLOv6-T*, the mAP^0.5:0.95 increased by 1.1% from 43.4% to 44.5% and the mAP^0.5 increased by 1.0% from 84.3% to 85.3%. This means that the proposed DePAN neck can improve the model performance in tumor detection on medical images.

Figure 6 gives the results of prediction by YOLov6-T and YOLOV6-T* on the breast CT medical images. Figure 6A are the labelled images of a benign tumor and a malignant tumor respectively. Figure 6B shows the results of prediction to the corresponding tumors by YOLOv6-T, and Fig. 6C provides the results of prediction to the tumors by YOLOv6-T with DePAN neck (i.e., VOLOv6-T*). It can be observed that the benign tumor was classified to be malignant one by YOLOv6 and the bounding box predicted by YOLOv6 for the malignant tumor was too large and not accurate enough. However, by using VOLOv6-T with the DePAN neck, the malignant tumor was classified correctly and the predicted bounding box was more accurate.

Figures 7 and 8 illustrate the prediction results on the PASCAL VOC2012 dataset. As can be observed, the predicted classes are highly accurate, and the bounding boxes effectively encapsulate the detected objects. The visualizations demonstrate the robustness and precision of our model in identifying and localizing various objects within the images. The clear delineation of the bounding boxes and the correct classification of the objects underscore the effectiveness of our approach in real-world scenarios, highlighting its potential for practical applications in object detection tasks.