Ship detection based on semantic aggregation for video surveillance images with complex backgrounds

Anchor box clustering based on the K-means method

The anchor mechanism used in Faster R-CNN is also employed in YOLOv2. Anchor boxes are a group of initial boxes with fixed sizes and aspect ratios. The selection of initial anchor boxes, whether they align with the characteristics of the detection objects or not, directly impacts the detection performance. In Faster R-CNN, the anchor boxes are manually set, which introduces personal errors. In YOLOv2, the K-means clustering method (Si et al., 2023) is used to perform clustering calculations on the ground truth boxes of the PASCAL VOC2007 dataset (Everingham et al., 2010) to obtain the optimal width, height, and number of initial anchor boxes. It should be noted that when using the Euclidean distance as a metric, larger bounding boxes may result in larger errors than smaller bounding boxes. However, in practice, we aim to obtain a high IoU value from the prior anchor box, which is not dependent on the size of the prior anchor box. Therefore, the distance metric is defined in Eq. (2).

(2) $$d(box,centroid)=1-IoU(box,centroid)$$where box indicates the ground truth box of the ship object, and centroid indicates the central anchor box of the cluster. In this study, the average IoU is used as the measurement for clustering. The average IoU is the maximum average value of the IoU between the ground truth boxes and the central anchor box. The objective function for the average IoU is expressed in Eq. (3).

(3) $$f=\arg max{\displaystyle \frac{\sum _{i=1}^k\sum _{j=1}^{q_k}IoU\left(box,centroid\right)}{q}}$$where q and k depict the total number of objects and clusters, respectively, and $q_k$ denotes the number of samples of the $k$ th cluster center. Since the size and number of anchor boxes obtained by clustering on the PASCAL VOC2007 dataset are not suitable for the ship detection dataset, the anchor boxes of the ship detection dataset are reanalyzed by clustering, and the average IoU curve with different clustering numbers is obtained, as shown in Fig. 2. This shows that as k increases, the average IoU changes increasingly slowly. When $k<5$, the average IoU increases faster. When $k>5$, the average IoU becomes stable. With increasing k, the risk of model overfitting also increases. Considering the recall and complexity of the detection model, $k=5$ is adopted in this study. On the one hand, it can accelerate the convergence of the proposed method in the training phase, and on the other hand, it can reduce the error caused by the anchor boxes. When k is equal to five, the width and height of the initial anchor boxes are (0.982, 0.457), (2.085, 0.831), (3.683, 1.396), (6.371, 1.998), and (8.849, 3.298). The width–height ratios of the anchor boxes are (2.15, 2.51, 2.64, 2.68, 3.19). The shape of the anchor boxes is mostly short and wide, which accords with the narrow and long characteristics of the ship object in the dataset.

Semantic aggregation module

The CNN extracts deep features that contain rich semantic information, giving it an advantage in predicting large objects. On the other hand, the CNN also extracts shallow features that have less semantic information but contain richer location information, which is advantageous for predicting small objects. In YOLOv2, the shallow high-resolution features undergo deformation using the reorg layer to better capture detailed information, and then they are fused with the deep low-resolution features to effectively detect objects using features from different layers, as depicted in Fig. 3A. However, YOLOv2 overlooks the incorporation of deep features with rich semantic information into shallow features, and it utilizes only a single-scale object detection layer, resulting in poor detection performance for small objects such as fishing boats. Therefore, in this article, we propose the semantic aggregation module, as shown in Fig. 3B, to enhance YOLOv2. By integrating deep features with shallow features, the model’s classification and positioning capabilities for ship objects are improved. It is worth noting that the notation “2 × upsampling” represents 2x upsampling. The specific procedure involves convolving the deep features with a 1 × 1 convolution operation and performing a 2x upsampling operation, which are then fused with the shallow features. In line with the correlation between feature maps, ship objects of various scales are detected from the fused features through lateral connections. To facilitate fusion, a 1 × 1 convolution operation is employed to adjust the channel dimension to a fixed number, and 2x upsampling is used to ensure consistent spatial resolution between deep and shallow features. In the enhanced YOLOv2 model, the semantic aggregation module is added to the 24th and 16th layers, as stated in Table 1.

Multiscale ship detection layer

The YOLOv2 detection network utilizes only a 13 × 13 object detection layer to predict class and location details. However, this limitation restricts the perception scope, making it prone to missing or misidentifying relatively small objects or objects with a small proportion of pixels in the image due to long-range shooting. To address this, we propose the addition of a 26 × 26 object detection layer, which, in conjunction with the 13 × 13 object detection layer, maximizes the utilization of multiscale features, leading to improved ship detection across various scales. To fully integrate deep semantic features with shallow fine-grained features that are abundant in location information and enhance the detection performance for small objects, we leverage the semantic aggregation module and feature fusion module of YOLOv2 to establish a 26 × 26 object detection layer.

Figure 4 presents the multiscale ship detection layer of the enhanced ship detection model used in this study. The process for establishing the 26 × 26 object detection layer is as follows: First, the output feature map in the 10th layer (see Table 1) is transformed into a 26 × 26 × 1,024 feature map using the reorg layer. Next, the 13 × 13 × 1,024 feature map from the 24th layer undergoes a 1 × 1 convolution (with 512 convolutional kernels) and 2x upsampling via the semantic aggregation module to generate a 26 × 26 × 512 feature map. These two 26 × 26 feature maps, along with the 26 × 26 × 512 feature maps obtained from the 16th layer, are fused to create a fusion feature map of size 26 × 26 × 2,048. Finally, the fusion feature map is subject to convolution (Convs in Fig. 4 implies the initial use of 3 × 3 convolution for cross-channel information fusion, followed by 1 × 1 convolution for dimension reduction of the feature map). The feature map with reduced dimensions is then utilized to predict the objects. Furthermore, the feature map of size 52 × 52 includes rich fine-grained features and important location information of small-size objects, allowing for enhanced detection precision of small ships.

The process for establishing the 13 × 13 object detection layer is as follows: The output feature map from the 16th layer (see Table 1) is subjected to a 1 × 1 convolution operation to acquire a feature map of size 26 × 26 × 64. This is subsequently reshaped into a 13 × 13 × 256 feature map using the reorg layer to leverage the fine-grained features of the model. Next, the output feature map from the 27th layer and the output feature map from the 24th layer are fused to obtain a feature map of size 13 × 13 × 1,280. Finally, cross-channel information fusion is performed using a 3 × 3 convolutional layer, followed by channel reduction through a 1 × 1 convolutional operation. The resulting feature map with reduced dimensions is employed for object prediction.

Due to the inherent characteristics of the feature maps, the feature map of size 26 × 26 employs two smaller anchor boxes ((0.982, 0.457), (2.085, 0.831)) and six categories, while the feature map of size 13 × 13 utilizes three larger anchor boxes ((3.683, 1.396), (6.371, 1.998), (8.849, 3.298)) and covers six categories.

Therefore, the network architecture of the proposed method is illustrated in Fig. 5. The input image is resized to a dimension of 416 × 416 pixels. Notably, the feature extraction network includes the initial 18 convolutional layers along with two additional convolutional layers consisting of 1,024 convolution kernels measuring 3 × 3 in YOLOv2. The detection network is composed of two parts, specifically, the object detection layers that utilize 13 × 13 and 26 × 26 feature maps. In our proposed network architecture, since there are only convolution layers and pooling layers, the input image size can be dynamically adjusted. For instance, when the input image is resized to 480 × 480 pixels, the 13 × 13 and 26 × 26 feature maps are transformed to 15 × 15 and 30 × 30, respectively. In Fig. 5, the notation “Conv 3 × 3, 32, 416 × 416 × 32” represents the presence of 32 convolution kernels measuring 3 × 3, resulting in an output feature map of size 416 × 416 × 32. Similarly, “Max pooling 2 × 2, 208 × 208 × 32” denotes the utilization of a pooling kernel of size 2 × 2, leading to an output feature map of size 208 × 208 × 32. In Table 2, semantic aggregation modules are incorporated at layers 24 and 16, while the feature fusion module corresponds to layers 10 and 16, as well as layers 16 and 24.

Using the DIoU to optimize the loss function

The calculation of the YOLOv2 loss function is based on the IoU (Girshick et al., 2014), which represents the ratio between the intersection and union of the predicted bounding box and the ground truth box. It can be expressed in Eq. (4).

(4) $$IoU={\displaystyle \frac{area\left(B_p\cap B_{gt}\right)}{area\left(B_p\cup B_{gt}\right)}}$$where $B_p$ represents the predicted bounding box, $B_{gt}$ represents the ground truth box, and $area(\cdot )$ denotes the area of the image region.

When there is no overlap between $B_p$ and $B_{gt}$, the IoU equals 0, which cannot reflect the distance between two boxes. The gradient of the loss function will be 0, and the network cannot be optimized. Moreover, if the intersection of $B_p$ and $B_{gt}$ are the same, the IoU is the same, but their alignment modes are different. The IoU cannot accurately reflect the intersection degree of $B_p$ and $B_{gt}$.

To address the shortcomings of the IoU, Rezatofighi et al. (2019) proposed the generalized intersection over union (GIoU) method, which can be expressed in Eq. (5).

(5) $$GIoU=IoU-{\displaystyle \frac{C-\left(B_p\cup B_{gt}\right)}{C}}$$where C represents the minimum closure region. The GIoU ranges from −1 to 1. More iterations are needed for convergence. Therefore, Zheng, Wang & Liu (2019) proposed the DIoU to directly minimize the normalized distance between the center points of $B_p$ and $B_{gt}$, which can be expressed in Eq. (6).

(6) $$DIoU=IoU-{\displaystyle \frac{{\rho }^2\left(b_p,b_{gt}\right)}{c^2}}$$where $b_p$ and $b_{gt}$ denote the center points of $B_p$ and $B_{gt}$, $\rho (\cdot )$ represents the Euclidean distance, and c represents the diagonal distance of the minimum closure region containing both $B_p$ and $B_{gt}$. The DIoU diagram is shown in Fig. 6. The DIoU is more suitable than the GIoU for regression of object boxes. For $B_p$ and $B_{gt}$ in the horizontal and vertical directions, the DIoU converges faster than the GIoU.

The environment of inland waterways is complex and diverse, leading to challenges in ship detection due to crowded river conditions. To enhance ship detection effectiveness, the proposed method incorporates a loss function designed using the DIoU. The expression of the loss function is shown in Eq. (7).

(7) $$\begin{array}{rl}{L}_{loss}=& L_{confidence}+L_{coord}+L_{groundtruth}\\ =& \sum _{i=0}^W\sum _{j=0}^H\sum _{k=0}^A{l}^{\mathrm{\prime }}{\lambda }_{noobj}\times (-b_{ijk}^o{)}^2+l{\lambda }_{prior}\times \sum _{r\in (x,y,w,h)}{(prio{r}_k^r-b_{ijk}^r)}^2\\ & +l_k^{truth}({\lambda }_{coord}\times \left(\sum _{r\in (x,y,w,h)}{(trut{h}^r-b_{ijk}^r)}^2\right)+{\lambda }_{obj}\times (DIo{U}_{truth}^k-b_{ijk}^o{)}^2\\ & +{\lambda }_{class}\times \left(\sum _{m=1}^{C_T}{(trut{h}^m-b_{ijk}^m)}^2\right))\end{array}$$where $L_{confidence}$ denotes the confidence error of the background, $L_{coord}$ represents the coordinate error of the anchor box and prediction bounding box, and $L_{groundtruth}$ represents the sum of the coordinate errors, confidence errors and classification errors of the prediction bounding boxes matched with each ground truth box. W and H are the width and height of the feature map, respectively. A is the number of anchor boxes corresponding to each grid. In Eq. (7), $i,j,k$ represent the row and column of the current target center and the category to which the current target belongs, respectively, $b_{ijk}^o$ indicates that there are no objects in the current grid, ${\lambda }_{noobj}$ represents the weight coefficient of no object, ${\lambda }_{prior}$ represents the weight coefficient of the anchor box, $prio{r}_k^r$ indicates the anchor box coordinates of class k, and $l^{\mathrm{\prime }}=1_{MaxDIoU<Thresh}$ represents the maximum DIoU between $B_p$ and $B_{gt}$. When the DIoU is less than the set threshold, it is marked as the background. $DIo{U}_{truth}^k$ indicates the DIoU of $B_p$ and $B_{gt}$, $l_k^{truth}=1_k^{truth}$ indicates that an object exists in the anchor box, ${\lambda }_{coord}$ denotes the weight coefficient of the coordinate error, $trut{h}^r$ represents the coordinates of $B_{gt}$, $b_{ijk}^r$ denotes the coordinates of $B_p$ of class k, and ${\lambda }_{obj}$ represents the weight coefficient of the object, and $Io{U}_{truth}^k$ depicts the IoU between $B_p$ and $B_{gt}$. In Eq. (7), ${\lambda }_{class}$ is the weight coefficient of the category, m indicates the category of the current object, $C_T$ denotes the total number of categories, $trut{h}^m$ represents the true category of the object, and $b_{ijk}^m$ represents the category of the predicted bounding box object.

Multiscale input training strategy

To enhance the robustness of the proposed method, a multiscale input training strategy is adopted. This strategy allows for dynamic adjustment of the input image size. Since the total downsampling step size is 32, input image sizes that are multiples of 32 are selected. The minimum input image size is 320 × 320 pixels, while the maximum is 608 × 608 pixels. During the training phase, after every 10 batches, the input image size is randomly selected from {320, 352, 384, 416, 448, 480, 512, 544, 576, 608}. The corresponding final detection output feature map has a size {10, 11, 12, 13, 14, 15, 16, 17, 18, 19}. For the same training model, test set images of different sizes can be evaluated. According to the reference (Redmon & Farhadi, 2017), the object detection precision increases with increasing test set image size, but the detection speed decreases. Therefore, in this study, the input image size for the test set is set to 480 × 480 pixels, considering both the detection precision and speed. Data enhancement methods such as hue adjustment, random flipping, and exposure variation were used during training to improve the generalization ability of the model.

Performance on the SeaShips dataset

To assess the performance of the proposed method, it is compared to Faster R-CNN (Ren et al., 2017), SSD (Liu et al., 2016), YOLOv2 (Redmon & Farhadi, 2017), method (Shao et al., 2020), method (Liu et al., 2021), FCOS (Law & Heng, 2018), method (Lin et al., 2020), YOLOv3-tiny (Wang & Wang, 2023), YOLOv4 (Wang & Wang, 2023), YOLOv5s (Wang & Wang, 2023) and YOLOv8s (Luo, Li & He, 2024) on the SeaShips dataset under the same experimental conditions. VGG16 is used as the feature extraction network for Faster R-CNN and SSD.

The detection performance is evaluated from both quantitative and qualitative perspectives.

Quantitative analysis

Table 4 presents a comprehensive comparison of various methods for the SeaShips dataset, including [email protected], AP for each ship type, and FPS. Notably, the Faster R-CNN and method (Lin et al., 2020) achieved the higher [email protected], reaching 90.33% and 89.90%, respectively. However, their FPS is only 6 and 5, which is lower than those of the other methods listed in Table 4. Although SSD and FCOS have shown some improvement in detection speed, they are unable to achieve real-time detection (which is typically considered to be at 24 FPS in object detection). Similarly, although YOLOv2 boasts a detection speed of 34 FPS, its [email protected] is 5.18% and 1.83% lower than those of Faster R-CNN and SSD, respectively. Additionally, in regard to the fishing boat category, the APs of YOLOv2 are 14.69% and 8.89% lower than those of Faster R-CNN and SSD, respectively. The reason for this disparity lies in the fact that the fishing boat is classified and positioned on the feature map after 32 rounds of subsampling, even though it only occupies a small portion of the original 1,920 × 1,080 pixel image (approximately 70 × 130 pixels). Consequently, YOLOv2 often fails to capture vital information pertaining to small ships, resulting in poor detection performance for these vessels. In contrast, our proposed method increases the [email protected] by 2.32% and 4.15% compared to SSD and YOLOv2, respectively. Notably, the proposed method shows notable improvements in AP for bulk ships and ore carriers, as these ships differ significantly from others due to their specific cargo—bulk goods and ore, respectively. Furthermore, the [email protected] of our proposed method also surpasses that of method (Shao et al., 2020), method (Liu et al., 2021), YOLOv3-tiny, YOLOv4 and YOLOv5s. In particular, the AP for fishing boats in our proposed method exhibited the most substantial increase, increasing by 10.98% compared to that in method (Shao et al., 2020). This enhanced performance can be attributed to two key factors. First, the addition of a semantic aggregation module improves the method’s ability to classify and locate ship objects. Second, a large-scale object detection layer is incorporated, enhancing the detection functionality for small ships. Moreover, our proposed method employs a loss function based on the DIoU, further enhancing the overall detection effectiveness. Further, by comparing AP, it can be seen that the proposed method can obtain good detection performance for different types of ships. Although the adoption of a multiscale object detection layer and the increase in parameters in our proposed method reduce the detection speed compared to YOLOv2, YOLOv3-tiny and YOLOv5s, it still maintains a speed that is 1.47 times faster than YOLOv4 and 2.33 times faster than FCOS. Therefore, the proposed method adequately meets the requirements for real-time detection. Compared with the YOLOv8s, [email protected] is slightly reduced in the proposed method, but the semantic aggregation module and multi-scale object detection layer are used in the proposed method to improve ship detection performance. Compared with the other existing ship detection techniques, the proposed method can effectively improve the precision of ship detection and ensure the detection efficiency. However, YOLOv8s has high requirements on hardware, requires a large amount of GPU resources and high memory of the graphics card, which increases the use cost and maintenance difficulty.

Table 5 shows the model size and GFLOPs of different methods for the SeaShips dataset. It can be seen that the GFLOPs of proposed method is smaller than that of Faster R-CNN, SSD, method (Lin et al., 2020), FCOS and YOLOv4. And the GFLOPs of the proposed method only increases by 3.39 and 4.34 compared to that in YOLOv2 and YOLOv8s, respectively. The model size of the proposed method is smaller than that of the method (Lin et al., 2020) and YOLOv4, and slightly larger than that of YOLOv2. In general, the proposed method meets the requirement of real-time detection, and it achieves better detection performance.

Figure 8 presents a comparison of the precision‒recall curves between the proposed method and YOLOv2 for six different ship types. It is evident from the graph that the proposed method achieves a higher recall for all six ship types while maintaining the same precision. Additionally, the area enclosed by the solid curve and coordinate axis is also larger for the proposed method, indicating its superior detection performance compared to YOLOv2. Furthermore, as depicted in Fig. 8B, the proposed method exhibits a relatively smaller improvement in the AP for container ships compared to YOLOv2. Consequently, the precision–recall curve appears to be closer for this specific ship type. However, overall, the [email protected] of the proposed method surpasses that of YOLOv2.

Qualitative analysis

According to the qualitative analysis illustrated in Fig. 9, the visual detection results of both the YOLOv2 method and the proposed method on selected ship samples from the SeaShips dataset are presented. Figures 9A, 9C, 9E, 9G, and 9I demonstrate the visual detection results of YOLOv2, while Figs. 9B, 9D, 9F, 9H, and 9J show the visual detection results of the proposed method. Comparing Figs. 9A, 9B, 9C, and 9D, it can be observed that the YOLOv2 method misidentifies a general cargo ship as a passenger ship when the ship and background exhibit similarities. Conversely, the proposed method avoids such misidentification by differentiating between similar backgrounds and ships. Contrasting Figs. 9E and 9F, it becomes apparent that when ship objects are partially obscured, YOLOv2 generates two detection results with significant overlap for the same object, with the positioning anchor being considerably larger than the ship object itself. On the other hand, the proposed method delivers more accurate ship positioning and exhibits improved detection effectiveness. Examining Figs. 9G and 9H, it is evident that YOLOv2 experiences positioning errors when detecting remote small fishing boats. Conversely, the proposed method excels in detecting such small fishing boats, resulting in enhanced confidence scores. In Figs. 9I and 9J, YOLOv2 erroneously detects a remote small fishing boat as a passenger ship. In contrast, the proposed method accurately detects remote small fishing boats, avoiding misidentification. Overall, the proposed method demonstrates its ability to detect ship objects more effectively, accurately position them, and prevent similar background misidentifications when there are occlusions between ship objects and small ships at a distance. However, as observed in Fig. 9J, the proposed method still exhibits some deviations in the positioning anchor box for smaller fishing boats. Future research will focus on further improving the detection performance for small-size ships.

Analysis of influencing factors

To analyze the influence of each improved module on the detection performance, experiments were conducted on the SeaShips dataset. The results are presented in Table 6. In the case of using the IoU loss function, compared with YOLOv2, the semantic aggregation module and multiscale object detection layer improved the localization and classification performance of ship objects, resulting in an increase in the [email protected] of 3.42%. By using the GIoU loss function, the ship detection performance improved to a certain extent, increasing the [email protected] from 88.57% to 88.76%. Furthermore, by using the DIoU loss function, the [email protected] further increased from 88.76% to 89.30%. These results indicate that the use of the semantic aggregation module and multiscale object detection layer can enhance the detection effect of ship objects, while the DIoU loss function, which is more consistent with the object anchor box regression mechanism, can further improve the detection performance.

Performance on the SeaShips_enlarge dataset

To further validate the detection performance, experiments were also conducted on the SeaShips_enlarge dataset.

Quantitative analysis

The results of the quantitative analysis are presented in Table 7, which lists the AP, [email protected], and FPS for the SeaShips_enlarge dataset. The [email protected] on the new test set is 89.10%, which is 0.73% higher than that on the SeaShips_enlarge dataset. The AP of each type of ship also increased to different degrees, indicating that the diversity of samples in the dataset increased after supplementing the ship images collected in the Yangtze River. As a result, the trained detection model obtained was more robust, leading to better detection performance.

Qualitative analysis

The qualitative analysis is depicted in Fig. 10, which displays the visual detection results of partial ship samples of the proposed method on the Seaships_enlarge dataset. The proposed method can correctly detect ship objects even under small-size and strong light conditions (Figs. 10A–10D). Additionally, ship objects can also be effectively detected when the objects are dense, the background is complex, or obstructions occur (Figs. 10E–10H).