IMC-YOLO: a detection model for assisted razor clam fishing in the mudflat environment

IAIFI module

Intrascale feature interaction (AIFI) module is a key component of Real-Time DEtection TRansformer (RT-DETR) (Zhao et al., 2023). It utilizes a multi-head self-attention module in the transformer and integrates position information into the input features using absolute position coding based on sine and cosine positional encoding. This absolute position coding can be described in the Eq. (1): (1)${\displaystyle \begin{array}{c}\hfill PE\left(x,2i\right)=sin\left(\frac{x}{T^{\frac{2i}{d}}}\right)\hfill \\ \hfill PE\left(x,2i+1\right)=cos\left(\frac{x}{T^{\frac{2i}{d}}}\right)\hfill \\ \hfill PE\left(y,2i\right)=sin\left(\frac{y}{T^{\frac{2i}{d}}}\right)\hfill \\ \hfill PE\left(y,2i+1\right)=cos\left(\frac{y}{T^{\frac{2i}{d}}}\right)\hfill \end{array}}$ where x and y denote the position indexes of the element in width and height respectively, d denotes the dimension of the embedding vector, i denotes the index of the dimension of the embedding vector, and T is the set temperature parameter. By specifying the value of T, the period of the trigonometric function in the above equation can be adjusted to regulate the coding granularity more flexibly. From the above equation, the embedding vector of position (x, y) can be obtained, and by concatenating the embedding vectors of x and y, the rich position information of different frequencies can be captured to form the final position encoding.

The absolute position encoding method in the AIFI module avoids the dependence on the convolutional kernel and achieves a more flexible approach to feature processing than methods that rely on convolutional operations to obtain contained position information. The internal processing flow of the AIFI module is summarized in the Eq. (2): (2)${\displaystyle Q=K=V=Flatten\left(F_{in}\right)}$ (3)${\displaystyle F_{out}=Reshape\left(Attn\left(Q,K,V\right)\right)}$ (4)${\displaystyle Attn\left(Q,K,V\right)=Softmax\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)\times V}$

where the input tensor F_in is subjected to the operation of reshaping, resulting in the generation of query vector Q, key vector K, and value vector V for multi-head attention computation. Subsequently, in the Eq. (3), the model captures positional dependencies and higher-level semantic information through the computation of multi-head attention. The self-attention mechanism is illustrated as shown in the Eq. (4): Q represents the query matrix, K denotes the key matrix, V stands for the value matrix, and d_k refers to the dimension of Q and K. The multi-head attention module is composed of multiple self-attention modules, which is given by the Eq. (5) and the Eq. (6): (5)${\displaystyle mAttn\left(Q,K,V\right)=Concat\left(hea{d}_1,\dots ,hea{d}_n\right)W^O}$ (6)${\displaystyle head=Attn\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right)}$

where the W^O, $W_i^Q$ , $W_i^K$ and $W_i^V$ is the projections parameter matrices. Following this, an inverse operation is applied to reshape the tensor back to its original shape, yielding the final output tensor F_out.

Skip connection is a widely used operation in modern convolutional neural networks and is usually limited to two of the feature fusion methods: concatenation and addition. However, these two feature fusion methods ignore variations in the features themselves, giving the same weight to two tensors with different content. Therefore, this study introduces the iterative Attentional Feature Fusion (IAFF) (Dai et al., 2021). The feature fusion method of IAFF is shown in Fig. 6. The Multi-Scale Channel Attention Module (MS-CAM) within IAFF enables channel attention at multiple scales by varying the pooling scale. Specifically, MS-CAM employs global average pooling in the global attention branch, and the two branches undergo two point-by-point convolutions, as well as intermediate activation and batch normalization layers, respectively, which are element-wise addition to obtain the fusion features through the broadcasting mechanism, and the Sigmoid activation function to obtain the channel attentions ranging from 0 to 1. For the original input tensor X and the output tensor Y after a series of transformations, the channel attention α output from the MS-CAM is used as the weight of X, and (1- α) is used as the weight of Y. X and Y are weighted and added element-wise. IAFF sequentially performs two rounds of the above operations, and with the high-quality fusion features generated in the first round as the initial features, the fusion weights in the second round are sufficiently improved, which in turn realizes the full context-awareness and the final output tensor Z is obtained.

To extract deep features rich in more advanced semantic information and at the same time to enhance the model’s representational capability, this study adopts the IAFF-based skip connection operation to transform the AIFI module. The modified module called IAIFI is combined with the CBS module to replace the SPPF module in YOLOv8 model. As shown in Fig. 7, Firstly, with the help of the additional CBS module, the model can further extract the features in the original backbone network. Subsequently, the IAIFI module relies on the multi-head self-attention mechanism of the high-level features to further enhance the interactions among the high-level features, to obtain the correlations among the conceptual entities in the image. This design approach, which focuses on establishing relationships between different regions within low-resolution feature maps, allows the model to learn abstract correlations of a single razor clam burrow relative to the entire image. This helps in distinguishing the regions containing razor clam burrows from similar but non-target regions, thereby reducing the false detection rate of razor clam burrows in practical applications. Finally, it relies on IAFF to perform skip connection operations on the input tensor and output tensor. The significance of this design strategy is to achieve a more rational fusion of semantically and scale-inconsistent features during skip connection operations.

Refactoring head network

The head network of the YOLOv8 model consists of three different scales of feature maps suitable for recognizing large, medium and small targets. However, for the object detection task, the pixels occupied by a single razor clam burrow in the input image in the context of a mudflat environment are extremely limited, resulting in the three basic feature map scales being difficult to meet the recognition and regression requirements of the task. Therefore, IMC-YOLO introduces a larger feature map scale by adding an Upsampling module and a Concat module, and realizes accurate recognition of small targets such as razor clam burrows by deep fusion of features with other scales.

In the original YOLOv8 model, the fusion of features at different scales relied on the PAN-FPN structure of the neck network. However, the head network of YOLOv8 does not fully utilize features at different scales as inputs to the head network. Therefore, in order to obtain feature information at different levels and granularities, as well as to mitigate the inconsistency between the four different feature map scales, the adaptively spatial feature fusion (ASFF) (Liu, Huang & Wang, 2019) method was used in this study to improve the head network in IMC-YOLO. The value of this strategy is that, according to different spatial locations and scales, the ASFF method can dynamically adjust the feature weights at each scale, thus achieving a balance between feature information of different scales and importance. Specifically, during razor clam burrows detection, variations in shooting distance and angle inevitably cause shifts in features of different scales, which can impact the importance of these features. To mitigate this issue, we choose to enable the network to implicitly learn the importance of features at different scales, achieving a certain level of alignment for these features. The ASFF method relies on multiple convolutional layers and the final SoftMax function to dynamically learn the feature weights at different scales and obtain the final feature map through weighted fusion. The specific feature fusion formula is as the Eq. (7) and the Eq. (8): (7)${\displaystyle y_{ij}^l={\alpha }_{ij}^l{x}_{ij}^{1\to l}+{\beta }_{ij}^l{x}_{ij}^{2\to l}+{\gamma }_{ij}^l{x}_{ij}^{3\to l}+{\delta }_{ij}^l{x}_{ij}^{4\to l}}$ (8)${\displaystyle {\alpha }_{ij}^l=\frac{e^{{\lambda }_{{\alpha }_{ij}}^l}}{e^{{\lambda }_{{\alpha }_{ij}}^l}+e^{{\lambda }_{{\beta }_{ij}}^l}+e^{{\lambda }_{{\gamma }_{ij}}^l}+e^{{\lambda }_{{\delta }_{ij}}^l}}}$ where $x_{ij}^{f\to p}$ denotes the feature mapping of the f-scale feature at position (i, j) with respect to the p-scale feature. ${\alpha }_{ij}^l$, ${\beta }_{ij}^l$, ${\gamma }_{ij}^l$, and ${\delta }_{ij}^l$ are the weights of the feature mappings of the different scales at position (i, j) learned by the SoftMax function in the Eq. (8), which take the value range of (0, 1), and ${\alpha }_{ij}^l$ + ${\beta }_{ij}^l$ + ${\gamma }_{ij}^l$ + ${\delta }_{ij}^l$ = 1. $y_{ij}^l$ is the final feature obtained by weighted fusion at position (i, j).

FasterNet block

In aquatic organism farming activities, models usually need to be deployed to mobile devices with limited resources, and in this application scenario, the size of the model will be very limited. Therefore, in order to improve the applicability of the model in the above cases, this study introduces the FasterNet Block (Chen et al., 2023), to replace the Bottleneck in the C2f module, which improves the feature extraction capability of the model while realizing a certain reduction in the size of the improved model.

FasterNet Block employs a new lightweight convolution called PConv, as shown in Fig. 8, which achieves more efficient feature extraction while reducing the model size. In a conventional convolution operation, all input channels are involved in the computation, which leads to a large number of redundant computations. To solve this problem, PConv performs the traditional convolution operation only on some of the consecutive input channels, leaving the rest of the channels unchanged. The reason this approach is similar to, but different from, the direct deletion of some of the channels is that it ensures that the feature information can flow through all the channels in subsequent operations. By reducing the involvement of redundant channels in the model, we have minimized the risk of overfitting that may arise from model complexity and alleviated the demand on storage capacity during actual fishing operations. As shown in Fig. 9, compared to Bottleneck, the FasterNet Block adds the PConv operation, a convolutional kernel size of 3 ×3, an MLP structure implemented by two point-by-point convolutional layers with intermediate activation and batch normalization layers, and a skip connection operation to improve the network performance for all C2f modules of different depths.

TBCAM

In the field of computer vision, the attention mechanism has been widely used in various tasks such as semantic segmentation, instance segmentation and object detection. Attention mechanisms are flexible in feature processing and enable the model to better focus on key regions in the image, thus improving the accuracy of image processing tasks. Currently, hybrid attention mechanisms such as CBAM (Woo et al., 2018), GAM (Liu, Shao & Hoffmann, 2021) and TripletAttention (Misra et al., 2020) emphasize simultaneous attention to channels as well as spatial regions. This hybrid attention mechanism utilizing channel and spatial information enables the model to better understand the semantic structure and spatial layout of images. Therefore, to improve the model’s ability to process image features in the mudflat environment, this study proposes a hybrid attention TBCAM whose complete structure is shown in Fig. 10.

For the channel level, in order to enhance the feature expressiveness of each channel, TBCAM performs alternating average pooling and maximum pooling on the height and width dimensions of the feature map, respectively. Compared with single average pooling or maximum pooling, the alternate pooling strategy yields more effective and critical information in the width and height dimensions. Subsequently, the obtained features are fed separately into a shared MLP to learn the complex relationships between the channels. By adding the two copies of the output features after the above process and using a Sigmoid activation function to learn the nonlinear feature representation capabilities, a representation of the weights of each channel is obtained. Eventually, these channel weights are multiplied element-wise with the original input features, which is the complete process of TBCAM at the channel level.

As for spatial dimensions, traditional spatial attention tends to focus only on the intrinsic connection of spatial dimensions, ignoring the potential relationship between channel dimensions and spatial dimensions. Therefore, to utilize this potential relationship, TBCAM borrows the three-branch structure in TripletAttention, which establishes inter-dimensional dependencies by performing a dimension-level rotation operation on each branch, thus capturing cross-dimensional interactions of input features. Specifically, three branches of TBCAM are used to construct the link between the height and width dimensions, the link between the channel dimension and the height dimension, and the link between the channel dimension and the width dimension. For the channel-level output, the tensor x ∈ R^C×H×W will be passed to each of the three proposed branches. In the first branch, the maximum and average values of the tensor along the channel dimension are first computed and concatenation are performed. Subsequently, the weight of each feature point in the feature map is obtained by a convolution operation with a convolution kernel size of 7 × 7 without changing the dimension of the feature map and using batch normalization and Sigmoid activation function. Finally, the final output of the first branch is obtained by multiplying element-wise these weights with the original feature map. For the remaining two branches used to construct the dependencies between the spatial and channel dimensions, it is necessary to permute the dimensions to obtain the tensor y₁ ∈ R^W×H×C and the tensor y₂ ∈ R^H×C×W, respectively, and then take the same operation as in the first branch and permute the dimensions with their original states. The final output tensor is obtained by element-wise addition of the output tensors of the three branches described above, followed by a simple averaging calculation.

Eventually, this study embedded the TBCAM in the FasterNet Block before making a skip connection operation to obtain the Three Branch Attention Faster Net (TBAFN) module as shown in Fig. 11, which led to the improved module called C2f_TBAFN. The TBCAM allows the model to focus more on key areas related to razor clam burrows in the image, effectively reducing the impact of background noise and interference. This not only enhances the model’s ability to identify the target but also improves its performance when handling images with varying scales and complex backgrounds.