Object-oriented building extraction based on visual attention mechanism

Construction of a knowledge base for the shadow object feature

Shadow is ubiquitous in high-resolution RSI and is one of the basic features of RSI. Shadow is caused by many factors such as illumination conditions, terrain undulation, ground cover, etc. Shadow affects the interpretation of ground objects and brings difficulties to ground object extraction, target recognition, and change detection. Shadows can be generated by specific features such as buildings, trees, mountains, etc. (Zhou & Fu, 2021; Mostafa, 2017).

In various types of RSI, shadows exhibit different features due to the presence of different types of ground objects. Moreover, certain ground objects possess spectral features that resemble those of shadows. Consequently, when relying solely on a single feature to determine a criterion for distinguishing shadows, these ground objects may be erroneously identified as shadows. Hence, it is imperative to consider multiple features when discerning between shadows and other ground objects. This necessitates the extraction of various features to construct an object feature knowledge base and establish a discriminant criterion that accounts for the influence of different ground objects. The features used for constructing this knowledge base are as follows.

• Brightness

• The brightness indicated the light intensity reflected by the ground objects received by the sensor. The shadow area does not receive direct sunlight, and its brightness mainly comes from the scattering light. Because the light intensity of the shadow area is much lower than that of the non-shadow area, the brightness of the shadow area is usually low.

• Hue

• Due to the influence of Rayleigh scattering, the blue light with shorter wavelength accounts for a large proportion of sky light, which leads to the shift of the spectrum in the shadow area to the blue band, thus the hue becomes higher.

• Ratio/difference between green band and blue band

• In the visible light band, water has strong absorption of sunlight, and its reflection is mainly in the blue band. However, due to the influence of algae chlorophyll in water, reflection will be formed in the green band. If the suspended sediment content in water is high, the spectral value will shift to the red and green bands. In RSI, water has high hue and low brightness. If only the hue and brightness features are judged, it may lead to false extraction. In addition, some dark green trees also show similar features in RSI.

• According to the features of these ground objects and shadows, the ratio or difference between green band and blue band can be used to distinguish them. The ratio or difference of these ground objects is generally greater than that of the shadow area.

• Convexity model

• From the perspective of brightness, the brightness of shadow areas in remote sensing image is low, and the brightness of surrounding objects is generally higher than that of shadow area, which constitutes a concave convex surface. From the perspective of hue, the hue of the shadow area is high, and the hue of the surrounding objects is generally lower than that of the shadow area, forming a convex surface.

• Area and shape

• Compared with lakes and other ground objects, the area of shadow area is relatively small. Compared with rivers, ditches and other ground objects, the shadow area is more compact in shape.

After establishing a knowledge base for shadow features, it is still necessary to determine the criteria for each feature. This article utilizes the maximum between-class variance method to adaptively select the threshold and establish the criterion for shadow objects across different features.

Shadow extraction method process

In RGB color space, the shadow region in remote sensing image has no obvious features in red, green, and blue bands. Overall, hue, saturation, and brightness models have more advantages in remote sensing image shadow extraction. To make full use of the brightness, hue, and other features of shadows in RSI, and to use the convex model criterion as an auxiliary criterion to constrain the segmentation object in the image segmentation process, this article uses HSV color space for object-oriented segmentation of images. This article utilizes the object-oriented image analysis method to analyze the features of shadow regions in high-resolution RSI. It also constructs a feature knowledge base of shadow objects and proposes an object-oriented shadow extraction method for RSI. This method is based on multi-scale segmentation in the HSV color space. According to the flowchart provided in Fig. 1, the specific implementation process of the method is as follows:

1. The input image undergoes a color space transformation, converting it from RGB color space to HSV color space.

2. Various features of input images are extracted from both the RGB color space and the HSV color space. These features are then combined, and the adaptive threshold is calculated using the maximum between-class variance method.

3. Using object-oriented multi-scale segmentation method, combined with convex model criteria to segment image.

4. Extract the area, shape and convex model of the image object in the segmentation results.

5. Combine the features obtained from step (2) and step (4) to construct the shadow object feature knowledge base.

6. By utilizing a knowledge base of shadow features, each image object in the segmentation results is individually assessed based on discriminant criteria to determine if it is a shadow. This process ultimately yields the extraction results.

The building salient index under the visual attention mechanism

To make full use of the color features of buildings in high-resolution RSI and form a greater degree of differentiation with other types of features, this article proposes a remote sensing image feature space transformation method that considers the features of buildings, which is used to describe the spectral features of buildings in high-resolution RSI. Combined with texture features, a building saliency index (BSI) is proposed.

• Architectural feature dimension

Buildings usually have high brightness in high-resolution RSI. Through the observation of a large number of buildings in high-resolution RSI, this article believes that the color of the building roofs in the image are mainly gray, blue and red, and green roofs are less common. Based on this feature, this article proposes the feature transformation formula of architectural feature dimension:

(1) $$f_1=2\times {\lambda }_1\times R+2\times (1-{\lambda }_1)\times B-G$$

(2) $$F_1=\{\begin{array}{l}{f}_{1}if{f}_1>0\\ 0if{f}_1<=0\end{array}$$

In the formula, R, G, and B are the DN values of the image in the red, green, and blue bands. ${\lambda }_1$ is feature weights, which need to satisfy ${\lambda }_1\in$ between 0 and 1. When the image is dominated by a reddish roof, ${\lambda }_1$ is greater than 0.5; when the image is dominated by blue roofs, ${\lambda }_1$ is less than 0.5; when the roof color in the image is gray, ${\lambda }_1$ equals 0.5. When the roof color in the image is unknown or there are multiple colors of the roof, Eq. (1) can be replaced by the following equation:

(3) $$f_1=2\times max(R,B)-G$$

• Vegetation feature dimension

Vegetation usually has green hue and high saturation in high-resolution RSI. Based on this feature, this article proposes the feature transformation formula of the vegetation feature dimension:

(4) $$f_2=\frac{G-{\lambda }_2\times R-(1-{\lambda }_2)\times B}{G+{\lambda }_2\times R+(1-{\lambda }_2)\times B}$$

(5) $$F_2=\{\begin{array}{cc}{f}_2\hfill & if{f}_2>0\hfill \\ 0\hfill & if{f}_2<=0\hfill \end{array}$$

The vegetation feature dimension highlights the vegetation features in the image, such as grassland, and cultivated land woodland, which can better distinguish high-brightness vegetation and buildings, as well as low-brightness vegetation and shadow. If the obtained RSI contain near infrared bands, the normalized difference vegetation index (NDVI) can be used to replace the vegetation feature dimension.

• Shadow feature dimension

Shadow has the features of low brightness and high hue in high-resolution RSI. Based on this feature, this article proposes the feature transformation formula of shadow feature dimension:

(6) $$F_3=\frac{H}{V}$$

H and V are hue and brightness components of the image in HSV color space respectively.

The transformed image feature space is comprised of three dimensions: normalized building feature dimension $F_1$, vegetation feature dimension $F_2$, and shadow feature dimension $F_3$. In this feature space, the article proposes a building spectral feature index to better highlight the features of buildings, while suppressing vegetation and shadow features:

(7) $$I_F=\frac{2\times F_1-F_2-F_3}{2\times F_1+F_2+F_3}$$

Considering the spectral and texture features of buildings in high-resolution RSI, this article proposes the building visibility index in high-resolution RSI:

(8) $$BSI=I_F\times BASI$$

The BASI is built-up areas saliency index that indicates the probability of a given location being within a building area (Shao, Tian & Shen, 2014).

The adjacency criterion of buildings and shadows

In RSI, buildings are often accompanied by shadows. High-resolution RSI provides clearer visibility of both buildings and their corresponding shadows compared to low-resolution RSI. Additionally, there is a topological relationship between buildings and their shadows in RSI. In the Northern Hemisphere, when studying a region, the sun’s orientation is typically from south to east or south to west. Consequently, the shadow region in the building area is adjacent to it in the direction of north to west or north to east. The utilization of topological relations between buildings and building shadows aids in differentiating buildings from objects that share similar spectral features, such as squares and roads. This article outlines the adjacent criterion for buildings and shadows as follows. For a building candidate object $Ob{j}_b$, there is an image object $Ob{j}_s$. If the following conditions are satisfied, $Ob{j}_b$ and $Ob{j}_s$ conform to the adjacency criterion, and object $Ob{j}_b$ can be judged as a building object.

(9) $$SHADOW(Ob{j}_s)=TRUE$$

SHADOW is a logical property defined on an image object. The above equation represents an object $Ob{j}_s$ as a shadow object.

(10) $$ADJ(Ob{j}_b,Ob{j}_s)=TRUE$$

ADJ is a logical attribute defined between any two image objects. The above equation indicates that there is an adjacent topological relationship between object $Ob{j}_b$ and object $Ob{j}_s$.

(11) $$\frac{1}{n_b}\sum _{(x,y)\in Ob{j}_b}^{}y-\frac{1}{n_s}\sum _{(x,y)\in Ob{j}_s}^{}y<0$$

$n_b$ and $n_s$ are the number of pixels contained in $Ob{j}_b$ and $Ob{j}_s$, respectively. The above equation indicates that the center of gravity of object $Ob{j}_b$ is lower than that of object $Ob{j}_s$ in the longitudinal direction. For the image products with coarse correction, the longitudinal axis in the image is usually in the north-south direction. The above equation considers that the adjacent shadow objects should be above the candidate building objects.

(12) $$\sum _{\genfrac{}{}{0ex}{}{(x,y)\in Ob{j}_b}{(x,y+1)\in Ob{j}_s}}^{}1>r\times (\sum _{\genfrac{}{}{0ex}{}{(x,y)\in Ob{j}_b}{(x-1,y)\in Ob{j}_s}}^{}1+\sum _{\genfrac{}{}{0ex}{}{(x,y)\in Ob{j}_b}{(x+1,y)\in Ob{j}_s}}^{}1)$$

The ratio parameter $r$ is greater than 0, and usually $r$ is equal to 1. The above equation indicates that in the adjacent edge points of object $Ob{j}_b$ and $Ob{j}_s$, the case that the edge point of $Ob{j}_b$ is close to the edge point of $Ob{j}_s$ is more than the case that the edge point of $Ob{j}_b$ is close to the edge point of $Ob{j}_s$. This article believes that under normal circumstances, the main direction of the adjacent edge of the building shadow and the building should be horizontal. The above equation can eliminate some pseudo-building objects generated by other ground object shadows. For most multi-story and middle-high buildings in the north-south direction, The above equation is applicable. For some buildings in the east-west direction, it can be extracted by setting the ratio parameter r less than 1.

Flow of building extraction method

According to the flowchart provided in Fig. 2, the specific implementation process of the method is as follows:

• 1. Remote sensing image preprocessing

For RSI with uneven illumination, a color consistency processing method is used for image preprocessing. For RSI with thin cloud effect, the uniform light method can be used for preprocessing. The remote sensing image obtained after preprocessing is clearer and the details are richer, which is conducive to the subsequent processing of images such as segmentation and feature extraction.

• 2. Main feature extraction

The RGB color space of the pre-processed RSI is converted to a new feature space using a method specifically designed for RSI, taking into account the unique features of buildings. This transformation allows for distinct features of buildings, shadows, and vegetation to be observed in different dimensions within the new feature space, which facilitates the subsequent extraction of shadows and buildings. Additionally, texture features and edge features of the image are extracted, resulting in a composite of the main features of the image, including building features, vegetation features, and shadow features.

• 3. Visual saliency computation

According to the features of building areas and buildings in high-resolution RSI, the top-down prior knowledge is introduced to construct the texture features of building areas, and the visual attention model based on information theory is used to calculate the original index BASI of building areas. Then, according to the main features extracted in Step (2), the spectral feature index of the building is calculated, and combined with BASI, the building visible index BSI is calculated.

• 4. Object-oriented multi-scale segmentation

Based on the object-oriented image analysis method, the object-oriented remote sensing image multi-scale segmentation method is used to segment the remote sensing image into the transformed feature space, and the multi-scale image object set is obtained.

• 5. Rough extraction of buildings

According to the segmentation results of step (4), select the appropriate scale, and use the following formula to count the building features one by one for the image objects of this scale.

(13) $$Q_{\mathrm{b}\mathrm{u}\mathrm{i}\mathrm{l}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}}\left(O_k\right)=\frac{1}{n_k}\sum _{(i,j)\in O_k}BSI(i,j)$$

$O_k$ is the kth image object obtained by segmentation, $Q_{\mathrm{b}\mathrm{u}\mathrm{i}\mathrm{l}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}}\left(O_k\right)$ represents the building features of object $O_k$, $n_k$ represents the total number of pixels contained in object $O_k$, and $BSI(i,j)$ represents the building salient index at the midpoint $(i,j)$. According to the statistical results of building features of all objects, whether the object is a candidate building object is judged by a given threshold. The maximum between-class variance method is used to obtain the adaptive threshold.

• 6. Building area extraction

The object-oriented building area extraction method under the visual attention mechanism was used to extract the building area of RSI in the transformed feature space according to the building area dominant index BASI calculated in Step (3) (Shao, Tian & Shen, 2014).

• 7. Shadow extraction

Using the object-oriented remote sensing image shadow extraction method, the shadow object feature knowledge base is established. According to the segmentation results of Step (4), the appropriate scale is selected to extract the shadow of the image and obtain the set of shadow objects.

When the solar elevation angle at the time of image acquisition is large, the shadow area is usually smaller than the building area, and then the shadow object can be extracted at a smaller scale. When the solar elevation angle at the time of image acquisition is small, the shadow area is similar is size to the building area, and the shadow object can be extracted at the same scale.

• 8. Fine extraction of buildings

The building object set $R_{building}$ should have the following relationship with the building area object set $R_{build-up}$:

(14) $$R_{building}\subset R_{build-up}$$

This article argues that buildings should be inside the built-up area. Using the adjacency criterion of buildings and shadows proposed in the thesis, and combining the inclusion criterion of buildings and building areas, the pseudo-building objects are eliminated and the building objects are obtained.

Finally, a morphological closed operation and an open operation are performed on the binary image of the extraction results to eliminate the “burr” in the extraction results and obtain the final building extraction results.

Shadow extraction experiments

To validate the effectiveness of the method, this article selected two high-resolution remote sensing images for shadow extraction experiments. Additionally, manual extraction of shadow regions from the remote sensing images was performed through visual interpretation and used as reference extraction results. Two contrasting experiments were conducted using the shadow extraction methods based on spectral ratio (Tsai, 2006) and continuous thresholding (Chung, Lin & Huang, 2009). In order to compare the differences in extraction results between pixel-level and object-level methods, the spectral ratio method was also applied in the HSV color space.

Experiment 1 selected an aerial remote sensing image of the Beijing region, as shown in Fig. 3A. The manually extracted reference shadow region is shown in Fig. 3B, with the black portion representing the shadow area. Figures 3C and 3D display the extraction results of the spectral ratio-based method and the continuous threshold-based method, respectively. Figure 3E illustrates the shadow extraction results of the proposed method in this article.

In the following experimental results, red represents true positives (TP), blue represents false positives (FP), and green represents false negatives (FN). TP, FP and FN are explained in the Evaluating Indicator section.

Comparing the extraction results of three methods, it can be observed that the methods based on spectral ratio and continuous thresholding fail to fully extract some building shadows, resulting in missed extractions. Additionally, these methods do not employ object-oriented image analysis methods, leading to the presence of salt-and-pepper noise in the extraction results. On the other hand, the method proposed in this article provides more complete extraction of shadow areas and successfully captures all shadows of multi-story buildings. However, the article’s method fails to accurately extract the shadows of a row of trees projecting onto the grass in the lower left area of the image. This is because the area of these tree shadows in the image is too small, and the article’s method tends to prioritize the accurate extraction of larger building shadows. The segmentation scale used in the method is larger than the size of the tree shadows, resulting in the tree shadows not being segmented as independent objects.

Experiment 2 selected a part of WorldView-2 satellite remote sensing image near Wuhan Wuhan University, as shown in Fig. 4A. The manually extracted reference shadow area is shown in Fig. 4B, where the black part represents the shadow area. Figures 4C and 4D show the extraction results of the method based on spectral ratio and the method based on continuous threshold, respectively. Figure 4E shows the shadow extraction results of the proposed method.

Comparing the extraction results of three methods, it can be observed that the method based on spectral ratio and the method based on continuous threshold mistakenly extract a portion of deep green trees as shadows. Furthermore, due to the lack of an object-oriented image analysis approach, these methods result in a salt-and-pepper effect. The method proposed in this article successfully extracts complete building shadows, but it fails to accurately extract some tree shadows. This is because sunlight penetrates through the trees to a certain extent, causing the shadow areas to shift towards the green band. Additionally, tree shadow areas are usually scattered and have smaller individual areas. The method in this article tends to accurately extract larger building shadows, and the chosen segmentation scale fails to segment some tree shadows as independent objects.

Building extraction experiments

Experiment 3 is an aerial remote sensing image of Beijing, as shown in Fig. 5A. The reference buildings artificially extracted by visual interpretation are shown in Fig. 5B, where the white part represents the building. Figures 5C and 5D show the extraction results using eCognition software and the extraction results of buildings using the proposed method. It can be seen from Fig. 5C that the extraction results of eCognition software misextract the spectral features of the bare land on the upper left side of the image, the road on the right lower side, and the small square in the lawn between buildings and other areas similar to buildings as buildings, and there is a serious misextraction phenomenon. In addition, the eCognition software failed to extract two low-rise buildings in the upper right side and the central region of the image, and failed to accurately extract the green part of the roof color of the building in the upper side of the image, resulting in serious missing extraction. It can be seen from Fig. 5D that the method in this article completely extracts 20 multi-story buildings with large scale and three low-rise buildings with small scale in the image, and extracts one building with very small scale in the lower area of the image (the pixel number is less than 100). In addition, based on the adjacent criterion of buildings and shadows and the inclusion criterion of buildings and building areas proposed in this article, the method eliminates the pseudo-building objects with similar spectral features, such as high brightness bare land, roads, cement ground. The extraction results are better than those of eCognition software. There are still some shortcomings in the extraction results of the experimental image, such as the failure to extract the shadowed part of the low-rise building in the middle of the image. In addition, for part of the green roof color on the building on the upper side of the image, the description of the spectral features of the building can be corrected according to the prior knowledge of the image.

Experiment 4 is a part of WorldView-2 satellite remote sensing image near Wuhan Wuhan University, as shown in Fig. 6A. The study area is the area with thin cloud cover under the original image. To eliminate the adverse effects of thin clouds on ground object extraction and enhance the details of buildings and shadows in the image, the image is homogenized before building extraction. The reference buildings manually extracted by visual interpretation are shown in Fig. 6B, where the white part represents the building. Figures 6C and 6D show the extraction results using eCognition software and the building extraction results using the proposed method, respectively. It can be seen from the figure that the proposed method completely extracts 16 multi-story buildings with large image scale and two low-rise buildings with small regional scale on the upper right side of the image. At the same time, based on the adjacent criterion of buildings and shadows, the proposed method eliminates the vacant land with similar spectral features in the middle region of the image. Figure 6C shows that the extraction results of eCognition software mistakenly extract the empty land similar to the spectral features of buildings in the central region of the image as buildings. In addition, eCognition software failed to extract two small-scale low-rise buildings in the upper right side of the image, and there was a phenomenon of missing extraction. In addition, due to the complexity of building roof detail information in Experiment 4’s image, the eCognition software failed to accurately extract some small targets on the roof, resulting in incomplete extraction of building objects, some building objects appeared as holes, or even fractured. Through comprehensive analysis, the overall effect of the proposed method on the extraction results of the two groups of experimental images is better than that of the eCognition software.

Evaluating indicators

• Classification indicators

TP represents the prediction result as a positive sample, which is also the actual positive sample. FP represents the prediction result as a positive sample, but the actual sample is negative. TN represents the prediction result as a negative sample, which is also the actual negative sample. FN represents the prediction result as a negative sample, but the actual sample is positive. P^′, P, N^′, N, $Total$ are explained as follows:

(15) $$P^{\mathrm{\prime }}=TP+FP$$

(16) $$P=TP+FN$$

(17) $$N^{\mathrm{\prime }}=FN+TN$$

(18) $$N=FP+TN$$

(19) $$Total=TP+FP+FN+TN$$

• Producer accuracy, user accuracy and overall accuracy

Producer’s accuracy (PA) represents the ratio of correctly classified pixels of a certain category in the ground truth. User’s accuracy (UA) represents the ratio of correctly classified pixels of a certain category to the total number of pixels classified as that category. Overall accuracy (OA) represents the ratio of correctly classified pixels to the total number of pixels in the image. According to the confusion matrix for binary classification, PA, UA, and OA of the construction area are defined as:

(20) $$PA=\frac{TP}{P}$$

(21) $$UA=\frac{TP}{P^{\mathrm{\prime }}}$$

(22) $$OA=\frac{TP+TN}{Total}$$

• Quantity disagreement and allocation disagreement

Quantity disagreement (QD) refers to the difference in quantity between the extracted results and the actual situation. Allocation disagreement (AD) refers to the difference in distribution between the extracted results and the actual situation, which occurs when the extracted results do not completely match the actual situation. Quantity disagreement and allocation disagreement are manifestations of overall error in terms of quantity and distribution, respectively. The sum of quantity disagreement and allocation disagreement represents the overall error.

In particular, for binary classification, the QD and AD can be calculated by the following equation:

(23) $$QD=\frac{\left|P-P^{\mathrm{\prime }}\right|+\left|N-N^{\mathrm{\prime }}\right|}{2\times \mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}$$

(24) $$AD=min\left[\frac{P-TP}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}},\frac{P^{\mathrm{\prime }}-TP}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}\right]+min\left[\frac{N-TN}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}},\frac{N^{\mathrm{\prime }}-TN}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}\right]$$

The overall accuracy (OA) has the following relationship with QD and AD:

(25) $$OA=1-(QD+AD)$$

Comparative analysis