Combining convolutional neural network with transformer to improve YOLOv7 for gas plume detection and segmentation in multibeam water column images

Dataset preparation

In this article, the raw data are obtained from a Kongsberg EM710 multibeam bathymetry system measurement with a beam opening angle of about 120°, and the Seafloor Information System (SIS) acquisition software is used for data acquisition and storage. For the measured multibeam water column data, after quantifying the grayscale values using the backscattering intensity information, 320 water column images containing gas plume targets are obtained. The dataset is then divided at a ratio of 8:1:1, and data augmentation operations such as translation, rotation, and MixUp are carried out. Ultimately, 1,920 images are generated for training, validation, and testing. To meet the data requirements of the YOLO format, we use Labelme software to create the dataset labels.

Experimental environment

We conduct our experiments on the Ubuntu 18.04.5 operating system, equipped with a V100 graphics card and an Intel Xeon Processor CPU with 10 virtual cores, along with 32 GB of graphics memory and 72 GB of RAM. The GPU is accelerated using CUDA 11.1, and all the network models are built using Python 3.8 and Torch 1.8.1 within Visual Studio Code 1.75.1. To highlight the effect of the models proposed in this article, we use the same hyperparameters for all networks, and the specific settings are shown in Table 1.

Selection of baseline network and comparison of backbone networks

In this article, the segmentation network of YOLOv7 is selected as the improved baseline network, and the segmentation network of YOLOv7 can achieve the detection and segmentation of the target simultaneously, which not only retains the advantage of fast inference in the YOLO series but also shares the same part of feature extraction and maintains the consistency of features. Compared with the network that performs detection and segmentation separately, the simultaneous detection and segmentation results have more comprehensive target information, achieving more accurate target localization and reducing the possibility of misdetection and missed detection. During the improvement process, we conduct a large number of ablation and comparison experiments and evaluate them using previously introduced metrics to comprehensively verify the superiority of the YOLOv7-CB network proposed in this article. To better adapt to the morphological characteristics of the gas plume target, in this article, we first prune for the backbone network. As shown in Fig. 5, we sequentially replace the ELAN structure in the backbone network with two layers of standard convolution. From Table 2, it can be seen that with the reduction of the ELAN structure, the network is sequentially improved in each scale metric. In terms of accuracy (as shown in Table 3), retaining only one ELAN structure in the deep network achieves performance beyond the baseline. In the detection task, YOLOv7-1ELAN increased by 1.5%, 1.7%, and 0.2% over baseline on F1, mAP50, and mAP50:95, and by 1.7%, 4.7%, and 0.6% in the segmentation task, respectively. For the module at the shallow layer of the network, the extracted features are mostly the underlying features, such as edges and textures. Ordinary convolution has a certain ability in extracting underlying features. Replacing the first three ELAN modules with ordinary convolution can remove redundant features and make the model focus more on semantic features with distinguishing degree, thus improving accuracy.

Design and analysis of the YOLOv7-CB network architecture

The ELAN structure achieves better expressiveness by aggregating different hierarchical features, but its expressiveness is still limited by the local context. Therefore, in this article, we use the proposed C-BiFormer module to replace the ELAN structure in YOLOv7-1ELAN by stacking, so that the network can do global modeling while acquiring local features, which has better interpretability compared with the ELAN structure. We design two network structures with different scales, as shown in Tables 4 and 5. In the first network structure, YOLOv7-CBs, we comprehensively outperform the baseline network in terms of scale and accuracy by stacking only four layers of C-BiFormer modules. The number of parameters, FLOPs, GPU memory and weight size are reduced by 45.5%, 32.7%, 25.2% and 45.3% respectively, and the FPS is increased by 2.3, with the network having faster inference. In the detection task, F1 has the same accuracy as the baseline, but in mAP50 and mAP50:95, it increases by 0.8% and 1.1% respectively. In the segmentation task, F1, mAP50, and mAP50:95 increased by 0.6%, 3.7%, and 1.4% respectively. The results show that using only four-layer C-BiFormer can adapt well to the morphological characteristics of the gas plume target, which is mainly attributed to the internal sparse attention, which can quickly exclude irrelevant regions, and the use of self-attention, which can focus more on the features related to the gas plume. Although similar to YOLOv7-1ELAN in terms of accuracy metrics, the four-layer C-BiFormer is smaller in terms of model size and is able to perform inference tasks more efficiently with limited resources. In the second network structure, YOLOv7-CBl, we stacked 12 layers of C-BiFormer modules, allowing the network to significantly outperform the baseline network in terms of accuracy. In the detection task, F1, mAP50, and mAP50:95 increased by 3.9%, 3.4%, and 1.6% respectively. In the segmentation task, F1, mAP50, and mAP50:95 increased by 5.4%, 7.9%, and 1.9% respectively. And in terms of the number of parameters, FLOPs, GPU memory and weight size, the 12-layer C-BiFormer is still reduced by 44.2%, 28.5%, 23.3% and 44.1% respectively. By stacking multiple C-BiFormer modules, the model can analyze input from multiple views and layers, learn more complex features and patterns, and make the overall model structure more flexible, retaining more design space and scalability. However, in terms of inference speed, the new model is slower, mainly because the long-distance dependencies in the feature maps need to take into account more relationships between regions, and the deeper the layers, the longer the inference time.

To further analyze the effectiveness of the proposed modules and the constructed model, we plotted the effective receptive field (ERF) heatmap (Ding et al., 2022) of the backbone in the model. As shown in Fig. 6, the dark regions represent the ERF size and pixel contribution levels. YOLOv7 exhibits a smaller ERF, with high-contribution pixels concentrated around the center and minimal edge influence. In contrast, YOLOv7-CBs and YOLOv7-CBl demonstrate significantly expanded ERFs, with YOLOv7-CBl showing more uniform distribution of high-contribution pixels, indicating stronger spatial information aggregation capabilities. Based on this, we conducted in-depth quantitative analysis. As shown in Table 6, “t” represents the threshold of contribution scores, indicating the minimum area coverage ratio of top t% high-contribution pixels in the final prediction results. The corresponding values represent the proportion of this minimum area to the entire image. As the threshold t increases, the area coverage ratios of YOLOv7 to YOLOv7-CBl sequentially increase, reflecting expanding ERFs. At t = 20%, YOLOv7-CBl achieves 15.1% coverage, enabling more effective integration of regional information for small object feature extraction. At t = 99%, the ratio reaches 97.8%, indicating that most image information contributes significantly to the final prediction, facilitating better large object information capture. In this article, C-Biformer module is used to build the model, which has a larger effective receptive field than ELAN module. With the increase of model depth, the receptive field gradually increases, which enhances the multi-scale feature extraction ability of the model.

Comparison of the proposed method with other self-attention modules

To verify the superiority of the C-BiFormer module in various aspects, we replace the C-BiFormer module with self-attentive modules such as ViT (Dosovitskiy et al., 2020), Swin transformer (Liu et al., 2021b), Conv2former (Hou et al., 2022), and Contextual Transformer (CoT) (Li et al., 2023a), respectively, based on YOLOv7-CBl. ViT and Swin transformer are purely self-attentive modules that require more hardware resources. As shown in Table 7, ViT and Swin transformer have relatively high requirements in terms of the number of parameters, FLOPs, and GPU memory. Conv2former, CoT, and C-BiFormer are all methods that use convolutional operations to simplify or simulate self-attention and therefore have a relatively low resource footprint. Among them, C-BiFormer has the fastest inference speed and the same lowest number of parameters, while CoT shows better results in terms of FLOPs and GPU memory usage, which is because CoT mainly uses a 3 × 3 convolution operation to encode the context of key points, an operation that only involves key points in local neighborhoods and does not need to operate on the whole input feature map, reducing the computation and reduces the memory usage. In terms of model accuracy, YOLOv7-CBl shows the best results in all metrics, as shown in Table 8. F1, mAP50, and mAP50:95 outperform the second-ranked CoT by 2.9%, 1.8%, and 1.3% in the detection task and by 2.8%, 1.7%, and 0.1% in the segmentation task. Although CoT also achieves relatively good results with dynamic sparse attention as the starting point of module design, the range of contexts acquired by the 3 × 3 convolution operation is still limited, so CoT performs slightly worse than C-BiFormer when dealing with targets with elongated morphology such as gas plumes. The low accuracy of the other modules is because ViT directly splits the feature map into fixed-size image patches during the input process without any inductive bias, resulting in a model that does not adapt well to the morphological features of the gas plume target. Swin transformer, although it adopts a hierarchical approach to capture the multi-scale features, for some small-size and feature-less gas plume targets, may not be able to capture enough detailed information. Conv2Former adopts static convolution to simulate self-attention but fails to achieve the adaptive function in self-attention well.

Comparison of the proposed method with other common attention modules

To compare the accuracy between self-attention and common attention, we conduct experiments using four types of common attention mechanisms: Squeeze-and-Excitation (SE) (Hu, Shen & Sun, 2018), Convolutional Block Attention Module (CBAM) (Woo et al., 2018), Efficient Multi-scale Attention (EMA) (Ouyang et al., 2023), and Normalization-based Attention Module (NAM) (Liu et al., 2021a). As shown in Table 9, conventional attention modules enhance the model’s sensitivity to different features by learning relationships between channels or spatial elements, thus improving performance. The related improvement methods are also focusing on weight computation and cross-dimensional interactions. Although they require lower computational resources, they still emphasize local information and are more suitable for specific tasks. These methods show lower accuracy in the experimental results of this study. On the other hand, C-BiFormer’s position invariance is more versatile, preserving not only local features but also achieving global information interaction through self-attention. Furthermore, common attention modules often suffer from information bottleneck problems when stacked repeatedly in the model, causing the model to overly focus on channel or spatial information while disregarding contextual information. The residual connections and LN within the C-BiFormer module effectively address this problem, resulting in overall higher performance.

Analysis of ablation experiments based on YOLOv7-CB

The bi-level routing attention in C-BiFormer retains the top k most relevant regions in each region after coarse screening of the partitioned regions. Therefore, the size of k determines the sparsity and computational overhead of the model. In this article, we adjust the size of k so that the model can strike a balance between the relevance of regions and computational overhead. As shown in Table 10, we set four different values of k = 1, 2, 4, 8 for the experiment. Although the overall result is greatly improved compared to the baseline, but when k = 1 or 2, the connection graph is too sparse to capture some important correlation information; when k = 8, the connection graph is more dense and can capture more correlation information. Good results are obtained in some metrics, but more computational resources are used. All things considered, we choose k = 4 as the final correlation value, so that the model can maintain a good result in both precision and computational overhead.

YOLOv7-CB consists of multiple C-BiFormer modules stacked on top of each other, and in the second ablation experiment, we mainly evaluate the effect of the number of module stacks on the model performance. As shown in Fig. 7, we set the number of modules from 1 to 24 layers and evaluate them every four layers to see the trend of the model on mAP50. With only one layer of the C-BiFormer module, the accuracy of the new model is below the baseline, but when adding up to four layers, the model already outperforms the baseline in both the detection and segmentation tasks and grows rapidly in the subsequent process. After stacking 12 layers, the model reaches maximum performance and begins to show performance saturation. As the number of modules increases, so does the depth of the network, and the model begins to reuse similar features, resulting in redundancy when information is propagated between different layers, and the performance of the model cannot be further improved. In addition, due to the attention collapse (Zhou et al., 2021) problem of deep Transformer, the model cannot distinguish between important and secondary information, and the long-distance dependent modeling ability decreases. The enhancement of the receptive field will introduce more background noise in the deeper network, which will interfere with the key feature extraction and lead to the reduction of accuracy. However, the downward trend is slow, and at 24 layers, the model accuracy is still higher than the baseline. The results show that YOLOv7-CB is highly flexible and can maintain good performance after stacking C-BiFormer modules several times.

Experimental analysis of the proposed method with other state-of-the-art networks

In comparison with other SOTA (state-of-the-art) models, we select some networks that can achieve both target detection and segmentation for comparison experiments, e.g., YOLOv5 (Jocher et al., 2022), YOLOv8 (Ultralytics, 2023), YOLOv9 (Wang et al., 2024), YOLO11 (Ultralytics, 2024), YOLACT (You Only Look At CoeffificienTs) (Bolya et al., 2019), Mask R-CNN (He et al., 2017), and Cascade Mask R-CNN (Cai & Vasconcelos, 2018). From Table 11, it can be seen that in the detection task, the YOLO series network shows higher accuracy. This is mainly due to the continuous iteration and optimization of the YOLO series, which introduces the most advanced network architecture and training techniques, along with data augmentation and sample balancing strategies. These factors enable the network to extract richer features, and the improved network performs the best in all detection metrics. However, in the segmentation task, the accuracy of mAP50:95 is lower than that of YOLACT, Mask R-CNN, and Cascade Mask R-CNN, which are specifically used for segmentation. The reasons are: the concept of ProtoNet is introduced in YOLACT to represent the feature prototypes of the objects and reduce the segmentation error, Mask R-CNN takes advantage of the precise feature alignment provided by ROIAlign, which is more helpful in capturing the boundary and shape of the target, while Cascade Mask R-CNN gradually screens candidate targets through a cascade structure so that each stage focuses on processing targets with different levels of difficulty, thus being able to maintain good accuracy despite high thresholds.

Detection and segmentation results of gas plume targets in multibeam water column images

At the end of the experiment, we select some plume target images with different scales and characteristics for instance testing, and make a more intuitive comparison of the performance gap between YOLOv7-CBl and other state-of-the-art (SOTA) models. As shown in Fig. 8, the first column shows clear plume targets, which exhibit higher detection accuracy and consistent segmentation results under the detection and segmentation of YOLOv7-CBl. The second column consists of small plume targets. Some models are unable to perform complete segmentation well, and in terms of detection, YOLOv7-CBl demonstrates higher detection accuracy. The plume targets in the third column are affected by sidelobe noise, resulting in being divided into two segments after imaging. However, YOLOv7-CBl is less affected by the noise, and it can still perceive the morphological features of the targets well, with higher accuracy and more complete segmentation results. The experimental results show that YOLOv7-CBl can be better adapted to the performance morphology of the gas plume target on the seafloor as well as to the noise defects in multibeam water column images, which improves the application of multibeam WCI in target detection and segmentation.