YOLOv8n-DDSW: an efficient fish target detection network for dense underwater scenes

C2f-DCN module

In the backbone of YOLOv8, feature extraction is primarily performed by the C2f module. However, underwater fish targets have substantial flexibility and morphological variability. Standard convolutions are only suitable for the detection of some static targets with regular shapes and low complexity. As a result, it is proposed that two DCN layers and one Deformable region of interest (ROI) pooling layer be introduced into the bottleneck structure of C2f. DCN can help the model better learn information such as the scale, background and deformation of a target. Deformable ROI pooling can learn offsets, expand the perception field of the model, and extract more complex features. The C2f-DCN module proposed in this article can adaptively learn targets of different sizes, better capture the features of the target, and show great robustness. The C2f-DCN structure is presented in Fig. 3.

DCN module

Conventional standard convolutions are arranged in a regular grid and sampled at regular grid positions in a feature map. In the DCN, however, the convolution kernel is not a fixed N × N grid (Dai et al., 2017). Sampling positions are no longer limited to regular grid positions. Instead, free sampling is performed by introducing offsets, which thereby expands the perception field, as shown in Fig. 4.

The standard convolution operation is generally divided into two steps: sampling the input feature map with a fixed grid and performing a weighting operation on the values of each sampling point. Its feature matrix formula is demonstrated in Eq. (1).

(1) $$y(p_0)=\sum _{k=1}^K{w}_k\cdot x(p_0+p_k).$$

DCN adds an offset to the standard convolution, as shown in Eq. (2).

(2) $$y(p_0)=\sum _{k=1}^K{w}_k\cdot x(p_0+p_k+\mathrm{\Delta }{p}_k)$$where $x$ and $y$ represent input and output feature maps, respectively; K and k stand for the total number of sampling points and enumerated sampling points, respectively; $p_0$ refers to the current pixel; $w_k$ denotes the projection weight of the kth sampling point; $p_k$ is the kth position sampled by the predefined convolution grid; and $\mathrm{\Delta }{p}_k$ means the offset that corresponds to the kth sampling position.

In Eq. (2), the pixel $x(p_0+p_k+\mathrm{\Delta }{p}_k)$ at the sampled position after the offset is usually implemented using a bilinear interpolation method.

(3) $$x(p)=\sum _{q}G(q,p)\cdot x(q)$$

(4) $$G(q,p)=\mathrm{g}(q_x,p_x)\mathrm{g}(q_y,p_y)$$

(5) $$g(a,b)=max\{0,1-|a-b|\}$$where $p$ is an arbitrary position in the region, $p=p_0+p_k+\mathrm{\Delta }{p}_k$; $q$ represents all the spatial positions in input feature mapping; $x(q)$ stands for the value of all the integer points on the feature map; G denotes a bilinear interpolation kernel function, which is a two-dimensional kernel that can be decomposed into two one-dimensional kernels.

The specific implementation process of deformable convolution is illustrated in Fig. 5. The left and right sides of the picture represent input and output features, respectively. First, the convolution preprocesses input features to generate offsets and modulations. After that, these offsets transform regularly distributed 3 × 3-pixel points into irregular, arbitrarily distributed 9-pixel ones. Next, the input image is sampled using the generated new pixel points to get the sampled feature map. Finally, the sampled feature map is multiplied with the convolution kernel element by element and summed to obtain the final convolution result.

Deformable ROI pooling module

The shape and size of the deformable ROI pooling window are dynamically adjusted according to the target shape and size for better adaptation. Its process is displayed in Fig. 6. First, the ROI is automatically generated and divided into t × t subregions. Subsequently, the offset of each subregion is calculated, followed by the adjustment of the sampling position according to the offset to obtain the feature value of the new subregion. Next, adaptive maximum pooling is performed on the sampled feature values of each new subregion to obtain a fixed-size output. At last, the pooling results of all new subregions are spliced in a particular order to obtain the final pooling result.

DPSE attention mechanism module

With the help of attention mechanisms, models can focus on targets and ignore background information, which improves their detection performance. Among attention mechanisms, the squeeze-and-excitation (SE) attention mechanism can adaptively adjust feature weights at the channel level to establish the interrelationship between channels and enhance critical features (Hu, Shen & Sun, 2018). Its structure is displayed in Fig. 7, where X and X′ represent input and output feature maps, respectively; C stands for the number of channels of the input feature map; H and W denote the height and width of the map, respectively. It mainly comprises two operations: squeeze and excitation. First, the squeeze operation employs global average pooling for converting the H × W feature layer into a 1 × 1 feature layer to obtain channel-level global features and the weights of each channel. Then, the excitation operation passes the 1 × 1 feature layer through two fully connected layers and normalizes it by use of sigmoid contrast weights to obtain channel weights. In the end, the feature map is used to multiply the output weight vector to get a feature map with weight information.

The squeeze operation takes advantage of global average pooling, which is formalized, as shown in Eq. (6):

(6) $$z_c={\mathbf{F}}_{sq}({\mathbf{u}}_c)=\frac{1}{H\times W}\sum _{i=1}^H\sum _{j=1}^W{u}_c(i,j)$$where $z_c$ represents the feature vector after the cth channel compression operation, ${\mathbf{F}}_{sq}$ stands for compression operation, and ${\mathbf{u}}_c$ denotes the channel of the input feature map. $u_c(i,j)$ refers to the feature value of the ith row and the jth column of the cth channel of the input feature map; H and W indicate the height and width of the feature map, respectively.

The excitation operation is shown in Eq. (7):

(7) $$S_C={\mathbf{F}}_{ex}(z_c,\mathbf{W})=\sigma ({\mathbf{W}}_2\delta ({\mathbf{W}}_1{z}_c))$$where, $s_c$ represents the feature weight of the cth channel after the excitation operation; ${\mathbf{F}}_{ex}$ stands for excitation operation; ${\mathbf{W}}_1$, ${\mathbf{W}}_2$ indicate the weight parameters of the two fully-connected layers; $\sigma$ denotes the Sigmoid activation function; and $\delta$ refers to the ReLU activation function.

The channel recalibration operation is shown in Eq. (8):

(8) $${\tilde{\mathbf{x}}}_c={\mathbf{F}}_{scale}({\mathbf{u}}_c,s_c)=s_c{\mathbf{u}}_c$$where, $\tilde{\mathbf{X}}=[{\tilde{\mathbf{x}}}_{\mathbf{1}},{\tilde{\mathbf{x}}}_2,\dots ,{\tilde{\mathbf{x}}}_C],{\mathbf{u}}_c\in {\mathbb{R}}^{H\times W}.{\mathbf{F}}_{scale}({\mathbf{u}}_c,s_c)$ represents channel-wise multiplication between $s_c$ and ${\mathbf{u}}_c$. $s_c$ and ${\mathbf{u}}_c$ denote the scalar and feature map, respectively.

The SE attention mechanism applies global average pooling, which averages the feature values of all channels and can retain more channel information. Nonetheless, average pooling causes the model to lose some essential local feature information, while max pooling can highlight critical local features. For this reason, the DPSE attention mechanism, which makes a combination of average and max pooling, is proposed. Average pooling highlights background information and outputs the average value of activations in the image, whereas maximum pooling highlights texture information and outputs the maximum value of activations. This model can pay attention to global and local information simultaneously, which thereby improves its performance and robustness. Figure 8 presents the structure of the DPSE attention mechanism.

Equations (9) and (10) show the forms of average pooling and maximum pooling, respectively.

(9) $${\overline{X}}_C=\frac{1}{H\times W}\sum _{i=1}^H\sum _{j=1}^W{u}_c(i,j)$$

(10) $$Ma{x}_c=Max{u}_c(i,j),(1\le i\le H,1\le j\le W).$$

The squeeze operation is formalized, as shown in Eq. (11):

(11) $$z_c={\mathbf{F}}_{sq}({\mathbf{u}}_c)={\overline{X}}_C+Ma{x}_c.$$

The excitation operation is shown in Eq. (12). A Sigmoid activation function is added after the double pooling operation to capture the result of double pooling.

(12) $$s_c={\mathbf{F}}_{ex}(z_c,\mathbf{W})=\sigma \left({\mathbf{W}}_2\delta ({\mathbf{W}}_1(\sigma z_c))\right).$$

The channel recalibration operation is formalized, as shown in Eq. (13).

(13) $${\tilde{\mathbf{x}}}_c={\mathbf{F}}_{scale}({\mathbf{u}}_c,s_c)=s_c{\mathbf{u}}_c.$$

The DPSE attention mechanism proposed in this article combines the merits of average and maximum pooling. This model retains more channel-level information while increasing attention to essential features, which leads to a better understanding of the correlation and importance between different regions.

Small detection module

Small object detection has always been a challenge because losing the characteristic information of small objects is easy. The original YOLOv8 algorithm has three detection heads detecting small, medium and large targets. The input size is a 640 × 640 feature map. After several downsampling operations, the feature map at the detection layers has a size of 20 × 20, 40 × 40 and 80 × 80, respectively. These sizes are utilized for detecting large targets above 32 × 32, medium targets above 16 × 16 and small targets above 8 × 8, respectively. However, the detection head of YOLOv8 still has limited detection capabilities for small targets. It fails to make full use of shallow feature maps with richer information about small targets. In some cases, small targets are not detected in practical underwater scene applications.

Therefore, a small target detection layer with a 160 × 160 feature map is added to detect targets above 4 × 4. Upsampling and feature stitching are performed again based on the original algorithm model to obtain more detailed feature information on small targets. Subsequently, this feature information keeps being passed to the other three scale feature layers along the path of downsampling. The small detection head not only enhances the ability of the network to fuse features but also provides more shallow feature and positioning information. The structure is illustrated in Fig. 9.

Wise-IOU

IOU-based loss functions are widely used in object detection tasks. YOLOv8 uses complete IOU (CIOU) for calculating the regression loss of the bounding box. This loss function evaluates the resemblance between two bounding boxes based on the position, size and angle differences between them and calculates positioning loss, as shown in Eq. (14).

(14) $$L_{\mathrm{C}\mathrm{l}\mathrm{O}\mathrm{U}}=1-\mathrm{I}\mathrm{O}\mathrm{U}+\frac{{\rho }^2\left(b^A,b^B\right)}{c^2}+\alpha \nu$$where $b^A$ and $b^B$ represent the centroids of prediction and real frames, respectively, $\rho$ stands for the Euclidean distance between both points, and $c$ denotes the diagonal length of the smallest outer rectangle of prediction and real frames; $\alpha$ refers to a balancing parameter. In addition, $v$ is used for computing the consistency of the aspect ratios of prediction and target frames.

However, CIOU uses a monotonic focusing mechanism that rules out the balance between simple and complex samples. When training samples contain low-quality targets, the detection performance of the model decreases. Therefore, this article introduces the Wise-IOU loss function with a dynamic non-monotonic focusing mechanism for balancing the samples, as shown in Eqs. (15) to (17).

(15) $$L_{\mathrm{W}\mathrm{I}\mathrm{O}\mathrm{U}}=r{R}_{\mathrm{W}\mathrm{I}\mathrm{O}\mathrm{U}}{L}_{\mathrm{I}\mathrm{O}\mathrm{U}},r=\frac{\beta }{\delta {\alpha }^{\beta -\alpha }}$$

(16) $$\beta =\frac{L_{\mathrm{I}\mathrm{O}\mathrm{U}}^{\ast }}{\overline{L_{\mathrm{I}\mathrm{O}\mathrm{U}}}}\in [0,+\mathrm{\infty })$$

(17) $$R_{\mathrm{W}\mathrm{I}\mathrm{O}\mathrm{U}}=\exp \left(\frac{{\left(x-x_{gt}\right)}^2+{\left(y-y_{gt}\right)}^2}{{\left(c_w^2+c_h^2\right)}^{\ast }}\right)$$where $L_{\mathrm{I}\mathrm{o}\mathrm{U}}\in [0,1]$ represents the IOU loss attenuating the penalty term for high-quality anchor frames; $R_{\mathrm{W}\mathrm{I}\mathrm{O}\mathrm{U}}\in [0,\exp ]$ stands for the Wise-IOU penalty term strengthening the loss of normal-quality anchor frames; the superscript * represents not participating in backpropagation; $\overline{L_{\mathrm{I}\mathrm{O}\mathrm{U}}}$ is a normalization factor standing for the sliding average of increments; $\beta$ refers to the degree of outliers, where smaller values imply higher-quality anchor frames.

Comparative experiment of attentional mechanisms

Different attention mechanisms were embedded into the network model for comparison, to verify the effectiveness of the DPSE attention mechanism. The results are shown in Table 3. The first group is the control group that used the original YOLOv8n algorithm. From the second to the sixth group, squeeze and excitation (SE), efficient channel attention (ECA), convolutional block attention module (CBAM), Biformer and DPSE mechanisms were added to YOLOv8n, respectively. The results show that the mAP50 values of SE and DPSE mechanisms increased by 1.1% and 1.9%, respectively, without increasing parameters. The ECA mechanism increased parameters slightly and mAP50 by 1.4%. The CBAM mechanism significantly increased parameters, but only improved mAP50 by 0.4%. The Biformer mechanism led to a substantial increase in parameters and a decrease in accuracy. It can be seen from these results that the DPSE attention mechanism proposed in this article has the best performance.

Comparison experiment of replacing different number of DCN

To verify the effect of DCNs on network performance, how replacing different numbers of DCNs in the bottleneck structure of C2f influences performance was investigated. The bottleneck structure includes two convolutional layers. Five groups were designed for this experiment. Group 1 serves as the control group and is composed of the original YOLOv8n network model. Groups 2 and 3 are based on the YOLOv8n model and take the place of the 1- and 2-layer convolution in the bottleneck, respectively. Group 4 is based on the model proposed in this article and replaces the 1-layer convolution. Group 5 is the model proposed in this article and replaces the 2-layer convolution. The experimental results are presented in Table 4.

As demonstrated in Table 4, both parameters and GFLOPs decrease with the increasing number of replaced DCNs. Compared to the model with only one layer of convolutional substitution, the model proposed in this article improves mAP50, precision (P) and recall (R) values by 0.7%, 0.8% and 0.7%, respectively. Consequently, this article replaces all two layers of standard convolution at the bottleneck with DCNs.

Comparative experiment of loss function

To verify whether the loss function improved is effective, comparative experiments were performed based on the improved model YOLOv8n-DCN-DPSE-Small detection proposed. The experimental results are shown in Eqs. (5) and (6) in Table 2. The regression loss in the process of training is presented in Fig. 15. It can be found that the loss function optimized converges faster, and the predicted value gets closer to the actual one.

Comparative experiment of different detection algorithms

The results in Table 5 indicate that the two-stage target detection algorithm Faster-RCNN has significantly larger parameters and GFLOPs but lower detection accuracy than single-stage algorithms. The YOLOv8n-DDSW algorithm proposed in this article outperforms single-stage algorithms such as selective synaptic dampening (SSD), YOLOv5s, YOLOv7 and YOLOv7-tiny in terms of detection accuracy and the calculated number of parameters. Compared to the original YOLOv8n algorithm, the YOLOv8n-DDSW algorithm improves mAP50, precision (P) and recall (R) values by 3.9%, 3.7% and 6.1%, respectively, while slightly reducing the number of parameters and computations. Comprehensive analysis shows that the algorithm presented in this article surpasses current mainstream target detection algorithms and possesses significant advantages in detecting underwater fish targets.