DGCFNet: Dual Global Context Fusion Network for remote sensing image semantic segmentation

CNN-based semantic segmentation methods

FCN (Long, Shelhamer & Darrell, 2015) is a CNN structure that solves semantic segmentation problems in an end-to-end manner. It uses transposed convolutional layers instead of fully connected layers in traditional image classification networks, and can gradually reduce the resolution of feature maps through stacked convolutional layers to obtain more semantic information. Afterwards, FCN-based methods dominated in the field of semantic segmentation of remote sensing images. However, due to the simplicity of its decoder, it is prone to misclassify image pixels, resulting in low segmentation accuracy. To address this issue, UNet (Ronneberger, Fischer & Brox, 2015) proposed an encoder-decoder network that utilizes skip connections to preserve the spatial information of the input image, allowing UNet to utilize semantic information at different levels while maintaining high-resolution features. Therefore, nowadays, the encoder-decoder structure has become the mainstream network for remote sensing images semantic segmentation. Nevertheless, due to the limitations of convolutional kernels, the limited receptive field can only extract local semantic features and lack the ability to global modeling of the entire image. To solve this problem, PSPNet (Zhao et al., 2017) constructed a feature pyramid through multiple parallel convolutional and pooling layers, which can receive receptive fields of different sizes to capture multi-scale contextual information. DeeplabV3 (Chen et al., 2018a) introduced an atrous spatial pyramid pooling module that captures multi-scale contextual information through multiple dilated convolutions with different dilation rates. In the field of remote sensing images, A²-FPN (Li et al., 2022) proposed an attention aggregation module that adaptively selects features at different scales through attention mechanisms, and combines feature pyramid networks to achieve effective learning and utilization of multi-scale features. MAResU-Net (Li et al., 2021a) proposed a linear attention mechanism and incorporated it into the skip connections of the encoder decoder structure to make the network automatically focus on important regions of the image, thereby improving segmentation accuracy. LANet (Ding, Tang & Bruzzone, 2020) proposed a local attention mechanism that introduces a local perception module to perceive the target scale changes, meanwhile, utilizes multi-scale feature fusion and fine-grained semantic segmentation techniques to improve segmentation accuracy and robustness. Li et al. (2021b) introduced multiple attention mechanisms to simultaneously focus on information from different aspects and scales of an image. Although the above methods have achieved significant segmentation results, they mainly obtain global information by aggregating local features extracted by convolutional kernels, rather than directly modeling global relationships. To address the issue of image feature loss, Wang (2023) introduced a new segmentation network, G-Lite-DeepLabV3. By combining group convolution with an attention mechanism, their model effectively handles complex semantic features and enhances the network’s response to important features. In order to distinguish between benign and malignant tissues in medical images, Damkliang et al. (2023) proposed an integrated method that combines a UNet-based architecture with attention gate units and residual convolutional units for prostate cancer (PCa) tissue analysis, achieving excellent results. Dong et al. (2024) proposed the Adaptive Adjacent Context Negotiation Network (A²CN-Net) to address issues related to large-scale changes and small-sized targets. This network utilizes a composite fast Fourier convolution module to enhance global context information and adaptively fuses multi-level features. Qu et al. (2024) proposed a deep multi-branch residual Unet with fused inverse weighting gated control. By introducing a deep multi-branch residual module that utilized convolution, detailed image features were extracted at deeper levels, the inverse weighting gated control module enhanced the diversity of upsampling information through counterclockwise horizontal transmission attention. Finally, at the highest level of the U-shaped encoder, a pyramid attention mechanism with different receptive fields was used to address the loss of pixel-level information leading to boundaries. Unlike these networks, our proposed model not only uses convolutional neural networks but also incorporates Transformers to further capture the global context information.

Transformer-based semantic segmentation methods

Transformer (Vaswani et al., 2017) was initially applied in the field of NLP. To apply it to image semantic segmentation, the image processing task is transformed into the sequence task of 1D image blocks. In this way, contextual information can be obtained through the powerful sequence-to-sequence modeling ability of Transformer structure. Most Transformer-based semantic segmentation networks still follow the encoder-decoder structure and can be roughly divided into two categories. The first category is pure Transformer methods. For example, SETR (Dosovitskiy, 2020) utilized the self-attention mechanism in Transformer to capture the dependency relationships between pixels in an image, thereby significantly improving the performance of semantic segmentation. However, the length of the input sequence is linearly related to the image resolution, which can lead to low efficiency in processing high-resolution images. Swin Transformer (Liu et al., 2021) proposed a window self-attention mechanism that divides an image into a series of small image blocks and only calculates the self-attention of the image region within the window. The Transformer structure has also demonstrated strong performance in remote sensing images. CGVT (Deng et al., 2023) used the crisscross attention mechanism to extract features with different scales at different levels. By combining the crisscross attention mechanism with the ViT model, it can improve the understanding of details and global structures, thus enhancing the accuracy of semantic segmentation. DCSwin (Wang et al., 2022a) adopted Swin Transformer as the encoder to extract contextual information and proposed a novel densely connected feature aggregation module and two attention blocks to connect four hierarchical Transformer features, thereby obtaining rich multi-scale information and context relationships. Another type is a hybrid network that combines the advantages of CNN and Transformer. For example, ST-UNet (He et al., 2022) constructed a dual encoder structure in parallel with Swin Transformer and CNN, and hierarchically integrated the global dependencies of Swin Transformer into the features of CNN through a relational aggregation module. Ding et al. (2021) proposed a wide context network that uses an adaptive perceptual context feature fusion mechanism to dynamically select and fuse features of different scales. BANet (Wang et al., 2021a) proposed a bilateral perception network that includes dependency paths and texture paths. The former extracted global contextual relationships through Transformer, while the latter extracted texture features through CNN. AerialFormer (Hanyu et al., 2024) employed a Transformer as the encoder and utilized multi-dilated convolutional neural networks (MD-CNNs) for the decoder. By considering both local and global context information, it achieved powerful representation and enabled high-resolution segmentation. Distinguished from these networks, our proposed model builds upon the use of Transformer and further investigates the integration of information across different layers by exploiting their inter-layer relationships, thereby enhancing the model’s segmentation performance.

Overall structure

As shown in Fig. 1, the proposed DGCFNet network consists of an encoder and a decoder. The encoder uses ResNet18 as the backbone network to extract multi-scale semantic features. It consists of four stages of Resblocks, and each stage comprises two 3 × 3 convolution layers. Except for the first stage that uses a max pooling layer, the other three stages change the resolution of the feature map through a 3 × 3 convolution with a stride of 2. Therefore, given an image with size of H × W four different sizes of feature maps are generated: H/4 × W/4, H/8 × W/8, H/16 × W/16, H/32 × W/32. The decoder includes a multi-scale Transformer block that obtains multi-scale contextual information, two redistribution weight blocks that efficiently fuse the obtained intra-layer and inter-layer information, and a Seghead that achieves pixel-level classification predictions. Specifically, in the decoder, a 1 × 1 convolution is first used to unify the channel dimensions of the four output feature maps of the encoder to 64, and then the corresponding feature images of the decoder are fused through skip connections. In this way, the fused features include local semantic features generated by the encoder and global semantic features generated by the decoder, which is beneficial for high-precision semantic segmentation of remote sensing images. At the same time, the feature map obtained from the last ResBlock is input into the multi-scale Transformer block, which undergoes a 3 × 3 convolution and a DGEM module to extract multiscale contextual information. In addition, inter-layer information and intra-layer information are fused through two redistribution weight modules, which is composed of a DGE module, a CII module, and a FIG module in sequence. The details of the proposed DGEM, CIIM and FIGM are following.

Dual-branch global extraction module

For complex remote sensing images, both local and global contextual information are crucial for semantic segmentation. Therefore, a dual-branch global extraction module is proposed to extract detailed local information and rich global information. which includes a global attention branch (GAB) and a global compensation branch (GCB). As shown in Fig. 2, the global attention branch uses a multi-head self-attention (MSA) with windows to capture contextual information, while the global compensation branch aggregates contextual information and local information extracted from multiple average pooling layers with different sizes.

Inspired by Wang et al. (2022b), the global attention branch first utilizes 1 × 1 convolution to expand the channel dimension of the input feature map by three times. Then, the cross window context interaction module (CWCI) is used to divide the windows into four parts: upper left, upper right, lower left, and lower right. The horizontal average pooling layer is used to establish the horizontal relationship between the windows, and the vertical average pooling layer is used to establish the vertical relationship between the windows. Based on this cross window pixel dependency, the relationship between any two windows can be established and then the global contextual relationships of image can be obtained. Specifically, given an input feature $X\in {\mathbb{R}}^{C\times H\times W}$, where C represents the channel dimension, H and W represents the height and width of the feature map, the output of global attention branch $X_{cwci}$ can be represented as:

(1) $$X_1=\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}\left(X\right)$$

(2) $$X_{cwci}=X_1+CWCI\left(\mathrm{M}\mathrm{S}\mathrm{A}\left(\mathrm{L}\mathrm{N}\left(X_1\right)\right)\right)$$where LN means layer normalization.

Meanwhile, to extract detailed local information, the global compensation branch consists of multiple average pooling layers with different sizes in parallel. Specifically, three average pooling layers $\mathrm{A}\mathrm{v}\mathrm{g}\mathrm{P}\mathrm{o}\mathrm{o}{\mathrm{l}}_{(5,2)}$, $\mathrm{A}\mathrm{v}\mathrm{g}\mathrm{P}\mathrm{o}\mathrm{o}{\mathrm{l}}_{(9,4)}$, $\mathrm{A}\mathrm{v}\mathrm{g}\mathrm{P}\mathrm{o}\mathrm{o}{\mathrm{l}}_{(17,8)}$ and one global average pooling layer are first used to extract multi-scale contextual information, where $\mathrm{A}\mathrm{v}\mathrm{g}\mathrm{P}\mathrm{o}\mathrm{o}{\mathrm{l}}_{(i,j)}$ represents an average pooling with size of i × i and stride of j. Then, upsampling operations are adopted to restore the four outputs to their original input size. The information from each sub-branch are also fuse with the original input information through summation operation. At the same time, in each branch, a 1 × 1 convolution is used to reduce computational complexity, and a 3 × 3 convolution is used to further feature learning. After information fusion, a 1 × 1 convolution is used to restore the original channel dimension, and residual connection is employed to preserve the input feature information. Finally, the representation capability can be further increased through a multilayer perceptron (MLP). In summary, the output of DGEM $X_{out}$ can be expressed as:

(3) $$X_{2,7}=\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}\left(X\right)$$ (4) $$X_{3,4,5,6}=\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{3\times 3}\left(X_2\oplus \mathrm{U}\mathrm{p}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\left(\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}\left(\mathrm{A}\mathrm{v}\mathrm{g}\mathrm{p}\mathrm{o}\mathrm{o}{\mathrm{l}}_{\left(\mathrm{i},\mathrm{j}\right)}\left(X\right)\right)\right)\right)$$ (5) $$X_{out1}=\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}\left(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left(X_2,X_3,X_4,X_5,X_6\right)\right)\oplus X_7\oplus X_{\mathrm{c}\mathrm{w}\mathrm{c}\mathrm{i}}$$ (6) $$X_{out}=X_{\mathrm{o}\mathrm{u}\mathrm{t}1}+\mathrm{M}\mathrm{L}\mathrm{P}\left(\mathrm{L}\mathrm{N}\left(X_{\mathrm{o}\mathrm{u}\mathrm{t}1}\right)\right)$$where $X_{2-7}$ represents the output of different sub-branches in the global compensation branch, and $\oplus$ represents the element-wise addition operation. Upsample denotes upsampling operation.

Cross-level information interaction module

In semantic segmentation, the feature maps at different stages contain feature information at different levels. Typically, only a few simple and similar modules are used within each stage to process features to obtain the local interaction However, considering local enhancement only is not enough. It is necessary to learn cross layer contextual interactions on adjacent layer features, and thus CIIM module is proposed to explore the impact of correlations between features at different levels on semantic segmentation.

In semantic segmentation, we conclude that the element-wise multiplication of two feature maps contains common information, which can highlight prominent features more. While the element-wise addition of the two feature maps allows information to be complementary without missing unique information, which is beneficial for refining objects. Based on this observation, the proposed CIIM module is shown in Fig. 3 and consists of two self-attention operations. Given inputs $f_1$ and $f_2$, they are resized to $f_1,f_2\in {\mathbb{R}}^{c\times h\times w}$ firstly by the bilinear interpolation. Then, the element-wise multiplication and addition of these two feature maps are represented as $f_m=f_1\otimes f_2$ and $f_s=f_1\oplus f_2$, respectively, where $\otimes$ represents the element-wise multiplication operation. In addition, to reduce computational costs, the channel dimensions of them are halved through 1 × 1 convolution, resulting in two new features $f_Q\in {\mathbb{R}}^{\left(c/2\right)\times h\times w}$ and $f_K\in {\mathbb{R}}^{\left(c/2\right)\times h\times w}$. Consequently, the self-attention weight can be expressed as:

(7) $$\mathrm{C}=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(f_Q⊛f_K)$$where Softmax ( $\cdot$) is the softmax activation function, and $⊛$ is matrix multiplication. In this way, the interdependence relationship between the coexistence information of $f_m$ and the comprehensive information of $f_s$ can be established.

Next, the generated self-attention weight matrix are multiplied with $f_{V1}$ and $f_{V2}$ to obtain two new feature maps $f_1^{\ast }\in {\mathbb{R}}^{c\times h\times w}$ and $f_2^{\ast }\in {\mathbb{R}}^{c\times h\times w}$, where $f_{V1}=\mathrm{c}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}⊛f_1$ and $f_{V2}=\mathrm{c}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}⊛f_2$ are both linear mappings of input $f_1$, $f_2$, Afterward, $f_1^{\ast }$ and $f_1$ are added through residual connections, and the same is done for $f_2^{\ast }$ and $f_2$ Finally, the fused feature $f_{12}\in {\mathbb{R}}^{c\times h\times w}$ is obtained through convolution and element-wise summation. Specifically, the calculation process of $f_{12}$ is:

(8) $$f_1^{\ast }\in f_{V1}⊛\mathrm{C}$$ (9) $$f_1^{\ast }\in f_{V2}⊛\mathrm{C}$$ (10) $$f_{12}=\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}\left(\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{3\times 3}\left(f_1^{\ast }\oplus f_1\right)\oplus \mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{3\times 3}\left(f_2^{\ast }\oplus f_2\right)\right)$$

The output of CIIM module $f_{12}$ inherits the properties of two adjacent input feature maps $f_1$ and $f_2$, which can further supplement the local enhanced features and is beneficial for simultaneously characterizing and recognizing different objects.

Feature interaction guided module

In the previous section, the dual branch global extraction module uses self-attention mechanism to capture the correlation between different pixels, which can capture long-distance relationships and global contextual relationships in the image. The cross layer information interaction module uses cross self-attention to establish correlations between different levels, which can better understand the relationship between different scales and semantic features, and improve the processing ability of details and local information. To further promote the fusion of output information from the above two modules and improve feature representation capabilities, the FIGM shown in Fig. 4 is proposed.

Given that the two inputs of FIGM module are $f_{12}$ (output of the CIIM module) and $f_2$ (output of the DGEM module), and the feature vectors of their corresponding pixels are $\overrightarrow{v_1}$ and $\overrightarrow{v_2}$ respectively. The output of FIGM $\overrightarrow{f_3}$ can be expressed as:

(11) $$\sigma =\mathrm{S}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}((w_1({\overrightarrow{v}}_1))\cdot ((w_2({\overrightarrow{v}}_2)))$$

(12) $$\overrightarrow{f_3}=\sigma \overrightarrow{v_1}+\left(1-\sigma \right)\overrightarrow{v_2}$$where Sigmoid ( $\cdot$) represents the Sigmoid activation function, “ $\cdot$” is a placeholder, used to indicate that the input to the Sigmoid function can be any real number. $w_1$ and $w_2$ represent the weights generated by 1 × 1 convolutions.

Datasets

Vaihingen dataset (ISPRS, 2019b): The vaihingen dataset is a regional dataset collected in Germany urban scenes. It consists of 33 high-resolution remote sensing images captured by drones, with an average size of 2,494 × 2,064 pixels. Each image in this dataset has a corresponding ground truth label, which includes six categories: The white, blue, cyan, green, gold, and red in the figure represent impressive surface, building, low vegetation, tree, car, clutter respectively. In this article, 16 images are selected for training, and the remaining images are used for testing. Each image is segmented into small blocks of 1,024 × 1,024 pixels to meet the experimental requirements. Similar to Li et al. (2022), Ding, Tang & Bruzzone (2020), the Vaihingen dataset only has a small number of clutter/background, so the results of clutter/background are not presented.

Potsdam dataset (ISPRS, 2019a): The Potsdam dataset is an aerial image dataset collected in the Potsdam urban scene in Germany, widely used in computer vision and remote sensing image processing. It contains 38 high-resolution remote sensing images, each with a size of 6,000 × 6,000 pixels. Similar to the Vaihingen dataset, it also contains six identical categories and doesn’t need to show the results of clutter/background. Meanwhile, each image is labeled with pixel-wise ground-truth annotation. We randomly select 22 images for training, two for validation, and 14 for testing. To meet the experimental requirements, the original image is cropped into small image blocks of 1,024 × 1,024 pixels.

BLU dataset (Ding et al., 2021): The BLU dataset is captured in Beijing by 21st Century Aerospace Technology Corporation using the Beijing-2 satellite. It contains 4 RGB optical remote sensing images, each with a size of 15,680 × 15,680 pixels. Each original image is overlapped and divided into 64 small images of 2,048 × 2,048 pixels. They also have the corresponding ground-truth labels and include six categories: background/barren, built-up, vegetation, water, Farmland, and road. We randomly select 196 images for training, 28 for validation, and 32 for testing.

Implementation details

Training settings: All experiments in this article are implemented using the PyTorch framework on NVIDIA Tesla A800 GPU. We adopt AdamW optimizer to optimize model parameters, set the initial learning rate to 6e−4, and adjust the learning rate using cosine strategy. For the Vaihingen and Potsdam datasets, the images are further randomly cropped into small blocks of 512 × 512, meanwhile, data augmentation is performed using random scaling and flipping techniques, with a batch size set to 16. For the BLU dataset, the images are randomly cropped into small blocks of 256 × 256, and data augmentation is also performed using random flipping and random cropping operations, with a batch size set to 32.

Loss function: To alleviate the problem of class imbalance caused by large differences in the number of targets in remote sensing images, the loss function used in this article is a combination of cross entropy loss $L_{ce}$ and dice loss $L_{dice}$ (Deng et al., 2023). The specific formula is as follows.

(13) $$L=L_{ce}+L_{dice}$$

(14) $$L_{ce}=-\sum _{i=1}^n{t}_i\log \left(p_i\right)$$

(15) $$L_{dice}=1-\sum _{i=1}^n\frac{2t_i{p}_i}{t_i+p_i}$$where $t_i$ represents the ground truth, and $p_i$ is the maximum probability of softmax for class i.

Evaluation metrics: To fairly compare the performance of various models, several commonly used metrics (Marmanis et al., 2018; Mou, Hua & Zhu, 2019) are adopted, including overall accuracy (OA), mean intersection over union (mIoU), and mean F1-score (mF1). Their calculation formulas are as follows:

(16) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\mathrm{T}\mathrm{P}/\left(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}\right)$$ (17) $$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\mathrm{T}\mathrm{P}/\left(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}\right)$$ (18) $$\mathrm{F}1=2\times \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}$$ (19) $$\mathrm{I}\mathrm{o}\mathrm{U}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}$$ (20) $$\mathrm{O}\mathrm{A}=\frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}$$where TP, FP, TN, and FN represent true positive, false positive, true negative, and false negative, respectively. OA represents the ratio of correctly predicted pixels to the total number of pixels. IoU is defined as the intersection and union ratio of predicted and ground-truth maps, and the average IoU of all categories is mIoU. The F1 score is the harmonic mean of Precision and Recall for each category.

Ablation study

To evaluate the effectiveness of each module in the proposed network, ablation studies are conducted on the Potsdam dataset in this section. The baseline model adopts U-Net network with ResNet18 as the encoder, and the decoder only uses convolution and upsampling interpolation operations.

(1) Effect of DGEM module: As shown in Table 1, after introducing the Transformer based DGEM module in the decoding section, the values of mean F1, OA, and mIoU increase by 0.33%, 0.57%, and 0.43%, respectively. Especially in the Building category, the F1 value increase by 1.00%, which proves the effectiveness of DGEM. Moreover, to further explore the necessity of global attention branch and global compensation branch in DGEM, the segmentation performance is further evaluated when the global attention branch GAB and global compensation branch GCB are used separately. As reported in Table 2, removing only the global compensation branch decrease the value of mIoU by 0.32%, while removing the global attention branch decrease by 0.30%. However, when both GAB and GCB are added simultaneously, the F1 values for Imp.surf. and Car decrease slightly compared to when only GAB is added. This is because GCB tends to favor more long-range context information, while GAB can obtain more detailed information. Consequently, the detailed information for Imp.surf. and Car is somewhat interfered with by GCB. The above results confirm that both branches are crucial.

(2) Effect of CIIM module: The addition of the CIIM model enables the model to simultaneously utilize contextual information from the current and adjacent layers. From Table 1, it can be seen that the CIIM model can improve F1 values for most categories. Meanwhile, mean F1 and mIoU values also show some improvement, reaching 90.79% and 83.36%, respectively. However, the results of Imp.surf. and Building have indeed declined, because we simultaneously incorporate the DGEM and CIIM modules with equal weights, this results in better outcomes in DGEM or CIIM needing to be equally weighted with the opponent, which notably causes information interference and limits the model’s performance. To further investigate the effectiveness of CIIM in different locations, the second and third DGEM are selected for ablation studies. As shown in Table 3, when the CIIM after the third DGEM is removed, mIoU value decreases by 0.16%. When removing the other one, mIoU value decreases by 0.21%. However, when both $f_{12}$ and $f_{23}$ are added simultaneously, the F1 scores for Tree and Car decrease slightly, indicating that information from different features can interfere with each other, leading to a decline in performance for certain categories. Nevertheless, when both $f_{12}$ and $f_{23}$ are added simultaneously, the F1 scores for Tree and Car decrease slightly, indicating that information from different features can interfere with each other, leading to a decline in performance for certain categories. The above results indicate the necessity of using two CIIM modules simultaneously.

(3) Effect of FIGM module: From Table 1, it can be observed that after adding the FIGM module, our DGCFNet model achieve the best results in all categories. Especially for the Building category, the final model improve by 1.16% compared to the baseline. Conversely, not using the FIGM module may result in a 0.35% reduction in mIoU, as these weights may be incorrectly assigned through pixel-wise addition. This indicates that the addition of feature maps may lead to information confusion, making it difficult to correctly learn specific semantic regions and boundaries.

The Baseline model will cause large areas of Imperfect Surfaces to be incorrectly segmented into Buildings. The addition of DGEM module significantly reduces erroneous segmentation, but inaccurate boundary segmentation is caused by only using multi-scale contextual information within the current layer. Then, by introducing inter-layer contextual information through the CIIM module, the boundary segmentation results are further improved. However, the fusion of contextual information from the current and adjacent layers through simple element-wise addition may lose spatial information, making it difficult for the model to accurately locate the boundaries of objects during prediction. By incorporating the FIGM module, the output boundaries of the proposed DGCFNet model become smoother and there is a significant reduction in erroneous segmentation.

Comparison with other methods

This section will compare the proposed DGCFNet with some existing methods, including FCN (Long, Shelhamer & Darrell, 2015), LANet (Ding, Tang & Bruzzone, 2020), A²-FPN (Li et al., 2022), MANet (Li et al., 2021b), MAResUNet (Li et al., 2021a), BANet (Wang et al., 2021a) and DCSwin (Wang et al., 2022a). Except for the BANet and DCSwin are methods based on Transformer structure, the others are all based on CNN. As with the experiments involving A²-FPN, BANet, MANet, we crop the original images and perform data augmentation by rotating, resizing, horizontally flipping, vertically flipping, and adding random noise.

(1) Results on Vaihingen dataset: The quantitative results of the comparative methods are shown in Table 4. We can see that the proposed method can achieve the highest values of meanF1, OA, mIoU, and the second highest F1 value. Compared with BANet, the meanF1 value of our method increases by 1.42%, OA increases by 0.70%, and mIoU increases by 2.20%. Meanwhile, the proposed method outperforms BANet in all categories, especially in the small target category Car, achieving an F1-score of 84.50%, which is 4.32% higher than BANet. The above results consistently prove that the proposed method not only has stronger global contextual modeling ability, but also has significant improvement in the segmentation of small targets.

Compared with other models, the proposed method effectively reduces segmentation errors, and has better segmentation performance at different scales. For example, in the large target category of Building, our method can more accurately segment the complete boundary of objects. The above results show that using a hybrid model of CNN and Transformer is feasible and can produce even better results.

(2) Results on Potsdam dataset: The quantitative comparison results of different methods on the Potsdam dataset are shown in Table 5. It can be seen that the method proposed in this article achieves the highest mean F1, OA, and mIoU, and only slightly lower F1 values than the MAResU-Net method in the small target category of Car. Compared with BANet, the proposed method improves the value of mean F1 by 2.24%, OA by 2.49%, and mIoU by 3.72%. Especially in the large target category of Building, the F1 value is 3.50% higher than BANet, indicating that using a CNN and Transformer hybrid model for spatial feature extraction in large-scale remote sensing images has a significant advantage in semantic understanding ability.

The edges of the “clutter/background” and “Building” classes have similar colors and are adjacent to the Imperfect Surface class. Therefore, the edges of the Building class are easily misclassified as the “clutter/background” class. The proposed method can help the model better cope with noise and interference in images by fusing feature maps at different levels, and obtain a rich global contextual relationship, enabling the model to recognize semantic information and suppress the influence of noise. Similarly, the small target category of Imperfect Surface is located near the Low Vegetation and Building, which can easily cause segmentation errors. Our model can identify the above regions well.

(3) Results on BLU dataset: The quantitative results of different methods on the BLU dataset are shown in Table 6. The results indicate that our method achieves the highest meanF1 and mIoU values. Moreover, in the Water category, the proposed method can achieve the highest F1 score, reaching 83.08%. Compared with BANet, the meanF1 of DGCFNet method increases by 3.18%, OA increases by 1.06%, and mIoU increases by 4.00%. Especially on the large targets such as Water, our method outperforms BANet by 6.03%, indicating that it can more comprehensively utilize the contextual information of images. However, other categories did not achieve the best results, except for the Water category. Due to its large area and relatively distinct characteristics, the Water category achieved good results that are significantly better than those of other categories. While the results for the other categories are slightly lower, this does not affect the final average result.

Since the Water class and the adjacent Farmland class share similar colors, other methods may mistakenly segment Water as Farmland due to insufficient discriminative information. Additionally, other methods are prone to significant omissions due to the limitations of their receptive fields. However, compared to other methods, our model, which introduces a hybrid structure of CNN and Transformer, achieves more complete and smoother boundary segmentation.

In the evaluation of network models, parameters and memory are crucial for assessing the network. We compared our DGCFNet with other segmentation networks on the Potsdam dataset, as depicted in Table 7.

Limitation

Although the proposed DGCFNet has better understood the relationship between different scales and semantic features by utilizing intra-layer and inter-layer information, there are still potential aspects that need to be considered.

Firstly, the total number of parameters in DGCFNet is 23.9M, which is smaller than that of medium-sized networks like DCSwin (45.6M) and MANet (35.9M), but larger than that of small-sized networks such as FCN (11.3M) and BANet (12.7M). Although DGCFNet has fewer parameters compared to DCSwin and MANet, it achieves a better IoU. However, while it achieves a higher mIoU than FCN and BANet, it consumes more computational resources. This indicates that our network structure does well in balancing accuracy and efficiency, but there is still room for reducing the model size in the decoder by employing model pruning techniques.

Secondly, although we have improved the segmentation capability for targets of different sizes by incorporating a global compensation branch in the Double-branch Global Extraction Module, there is still significant room for improvement in segmenting targets within complex backgrounds. Therefore, our future work will focus on model lightweighting and enhancing the capability to segment small targets.