Object-based multiscale segmentation incorporating texture and edge features of high-resolution remote sensing images

Multiscale image segmentation with color and shape features

MSS is used in the famous remote sensing image analysis software eCognition Developer, which is produced by Definiens (Munich, Germany) and serves as the core algorithm for object-based image analysis. The basic idea of MSS is that images in nature conform to the fractal theory: that is, a typical structure will appear on different scales of an image, and show a certain degree of irregularity and self-similarity on various scales. The fractal feature is particularly obvious in remote sensing images. Mountains and rivers in the images all have fractal features. For example, in remote sensing images of urban areas, dense building areas will present: at a fine scale (small scale), buildings and building shadows form light and dark texture features; at a medium scale, buildings and the open space between the buildings form a texture feature of light and dark; and at a coarse scale (large scale), the regularly arranged buildings and buildings also form a periodic texture feature. This self-similarity feature is manifested in the fractal feature of remote sensing images.

In many different segmentation scales, the MSS yields the segmentation result of the image at the corresponding scale, i.e., the image object collection. Connecting image objects of different scales in the same area in the image forms a hierarchical network of image objects, as shown in Fig. 1. In the hierarchical network, each image object not only saves its attribute information, such as spectral mean, variance, area, perimeter, edge, and other information, but also saves its adjacent object information and affiliation (child object and parents object). When comparing two image objects with similar attributes in a remote sensing image, it is particularly important to analyze their scale level and the semantic information between adjacent objects and subordinate objects.

The MSS is a bottom-up region merging algorithm. The algorithm adopts the merging criterion of the least increase in heterogeneity as the regional merging strategy. At the beginning of the segmentation, the algorithm regards each pixel in the image as a minimum image object and finds the local best mutual matching object for merging through the corresponding merging criteria. The algorithm regards the heterogeneity growth threshold as the scale parameter in image segmentation and, by setting different scale parameters, the image can be segmented at multiple scales.

In the MSS, the degree of object heterogeneity represents the degree of heterogeneity of the internal attributes of an image object. The algorithm uses the growth value of the object’s heterogeneity and its threshold as the criteria for judging whether the object is merged and with whom. The measurement method of image object heterogeneity is the core of the whole algorithm. The algorithm considers the spectral feature and shape feature of the image and proposes two standards for measuring the heterogeneity of image objects: spectral heterogeneity and shape heterogeneity. Spectral heterogeneity describes the degree of heterogeneity of the spectrum inside an image object. The definition of spectral heterogeneity h_color is as follows: (1)${\displaystyle h_{\text{color}}=\sum _c{\omega }_c\cdot {\sigma }_c.}$

c is spectral channel and ω_c represents weighting factor of c-th band, which is supposed to meet 0 ≤ ω_c ≤ 1 and ∑_cω_c = 1.σ_c represents the standard deviation of spectral values in the c-th band. For the shape feature of the remote sensing image object, the algorithm proposes two criteria for shape heterogeneity: compact heterogeneity and smooth heterogeneity. Compact heterogeneity describes the compactness in the shape of image objects. The closer the shape of the object is to a circle, the greater the compactness and the smaller the value of the compact heterogeneity. h_cmpct can be defined as follows: (2)${\displaystyle h_{\text{cmpct}}=\frac{l}{\sqrt{n}}.}$

n is the area of the image object (number of pixels), and l is the perimeter of the image object. Smoothing heterogeneity describes how smooth the image object is in shape. The closer the shape of the object is to a rectangle, the smoother the edge of the object, and the smaller the value of the smoothing heterogeneity. h_smooth can be defined as follow: (3)${\displaystyle h_{\text{smooth}}=\frac{l}{b}.}$

In the formula, l represents the perimeter of the image object, and b represents the perimeter of the smallest circumscribed rectangle of the image object. In raster images, the algorithm believes that the smallest bounding rectangle can be replaced, approximately, by the bounding box of the object. Therefore, when the calculated image object Obj1 and the image object Obj2 are merged into the image object Obj_Merge, the algorithm defines the growth value f of the overall object heterogeneity as the weighted sum of the spectral heterogeneity growth value Δh_color and the shape heterogeneity growth value Δh_shape: (4)${\displaystyle f={\omega }_{\text{color}}\cdot \Delta h_{\text{color}}+{\omega }_{\text{shape}}\cdot \Delta h_{\text{shape}}.}$

ω_color and ω_shape are the spectral weighting factor and shape weighting factor, respectively, which must meet 0 ≤ ω_color ≤ 1, 0 ≤ ω_shape ≤ 1 and ω_color + ω_shape = 1. The growth value Δh_color of spectral heterogeneity is defined as: (5)${\displaystyle \Delta h_{\text{color}}=\sum _c{\omega }_c\left(n_{\text{Merge}}\cdot {\sigma }_c^{\text{Merge}}-\left(n_{\text{Obj1}}\cdot {\sigma }_c^{\text{Obj1}}+n_{\text{Obj2}}\cdot {\sigma }_c^{\text{Obj2}}\right)\right).}$

In the formula, c represents the band, and ω_c represents the weighting factor of the c-th spectral band, which must meet 0 ≤ ω_c ≤ 1 and ∑_cω_c = 1.n_Obj1, n_Obj2, n_Merge is the area (that is, the number of pixels) of the object Obj1, the object Obj2, and the combined object. ${\sigma }_c^{\text{Obj1}}$, ${\sigma }_c^{\text{Obj2}}$, ${\sigma }_c^{\text{Merge}}$ is the standard deviation of the spectral value of the object Obj1, the object Obj2, and the combined object in the c-th band. The algorithm defines the shape heterogeneity growth value Δh_shape as the weighted sum of the compact heterogeneity growth value Δh_cmpct and the smooth heterogeneity growth value Δh_smooth: (6)${\displaystyle \Delta h_{\text{shape}}={\omega }_{\text{cmpct}}\cdot \Delta h_{\text{cmpct}}+{\omega }_{\text{smooth}}\cdot \Delta h_{\text{smooth}}.}$

ω_cmpct and ω_smooth represent the compact weight factor and smooth weight factor, respectively, which must meet 0 ≤ ω_cmpct ≤ 1, 0 ≤ ω_smooth ≤ 1, and ω_cmpct + ω_smooth = 1. Compact heterogeneity growth value Δh_cmpct and smooth heterogeneity growth Δh_smooth are defined as: (7)${\displaystyle \Delta h_{\text{cmpct}}=n_{\text{Merge}}\cdot \frac{l_{\text{Merge}}}{\sqrt{n_{\text{Merge}}}}-\left(n_{Obj1}\cdot \frac{l_{Obj1}}{\sqrt{n_{Obj1}}}+n_{Obj2}\cdot \frac{l_{Obj2}}{\sqrt{n_{Obj2}}}\right)}$ (8)${\displaystyle \Delta h_{\text{smooth}}=n_{\text{Merge}}\cdot \frac{l_{\text{Merge}}}{b_{\text{Merge}}}-\left(n_{Obj1}\cdot \frac{l_{Obj1}}{b_{Obj1}}+n_{Obj2}\cdot \frac{l_{Obj2}}{b_{Obj2}}\right).}$

In the above two formulas, n_Obj1, n_Obj2, n_Merge is the area (the number of pixels) of the object Obj1, the object Obj2, and the combined object, respectively. l_Obj1, l_Obj2, l_Merge is the perimeter of the object Obj1, the object Obj2, and the merged object, respectively. b_Obj1, b_Obj2, b_Merge is the perimeter of the smallest bounding rectangle of the object Obj1, the object Obj2, and the merged object, respectively. It can be seen from the above description that the image segmentation result obtained by the MSS meets the minimum sum of the weighted heterogeneity of the image object on a certain scale s. At this scale, the image objects in the segmentation result are small enough in attribute differences, and the attribute differences between image objects are large enough.

Multiscale segmentation incorporating texture feature

The object-based image analysis method takes the image object as the basic unit. The segmentation results of images at different scales are obtained through the bottom-up region merging process. The segmentation process considers the spectral and shape features of the image object. The texture homogeneity between adjacent pixels in the image is not considered. Based on the analysis image texture, we propose a remote sensing image texture feature descriptor based on time-frequency analysis. Furthermore, this section constructs the texture heterogeneity measure of the image object based on this method, and an MSS algorithm for object-based remote sensing images combined with texture feature is proposed.

Texture feature descriptor

When humans observe an image, they will find that although some ground objects do not have regularity in a local area, they will show a certain law when viewed as a whole. This feature of local irregularity and overall regularity is called texture. Humans have a clear perceptual understanding of the texture feature in images, but it is difficult to determine a unified mathematical definition. Currently, the commonly used methods of texture feature description can be divided into four categories: statistics-based methods, structure-based methods, model-based methods, and time-frequency analysis-based methods (Zhang & Tan, 2002).

Scholars have successively proposed methods to description texture feature based on time-frequency analysis. The most famous is the Laws texture (Laws, 1980). The basic idea is to first use multiple small convolution templates to filter the image to obtain features such as horizontal edges, high-frequency points, and vertical edges, and then use a larger moving window to convolve to obtain local texture energy. In terms of time-frequency analysis, after wavelet, wavelet tree and Gabor wavelet, nonsampled contourlet transform (NSCT) has gradually become an area of research interest. As shown in Fig. 2, the process is mainly divided into two steps: First, the image is divided into low-frequency subbands and multiscale high-frequency subbands with a nonsampled pyramid structure, and the multiscale decomposition results are obtained. Then, a nonsampled directional filter bank is used to decompose each high frequency subband in multiple directions. Since the filters used in the two steps are all nonsampled, the translation invariance of the filtering result is ensured.

We use nonsampled contourlet transformation to decompose the image to obtain multiscale and multidirectional subband f_{s, d}, where s represents the scale and d represents the direction. Then, we use the local texture energy function to calculate the local energy of each scale and direction subband, and use the result as the texture feature of the image. Taking the window size as (2n + 1) × (2n + 1), the calculation formula for the texture feature E_{s, d} at the coordinate (x, y) in the image is as follows: (9)${\displaystyle E_{s,d}\left(x,y\right)=\frac{1}{{\left(2n+1\right)}^2}\sum _{i=x-n}^{x+n}\sum _{j=y-n}^{y+n}\left|\right|f_{s,d}\left(i,j\right)\times g\left(i,j\right)|-e_{s,d}\left(x,y\right){|}^2}$ (10)${\displaystyle e_{s,d}\left(x,y\right)=\frac{1}{{\left(2n+1\right)}^2}\sum _{i=x-n}^{x+n}\sum _{j=y-n}^{y+n}|f_{s,d}\left(i,j\right)\times g\left(i,j\right)|}$ (11)${\displaystyle g\left(i,j\right)=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{{\left(i-x\right)}^2+{\left(j-y\right)}^2}{2{\sigma }^2}}.}$

In the above formula, we believe that in a partial window, the contribution of NSCT coefficients to the texture feature of the center of the window is inconsistent. The farther from the center of the window, the smaller the contribution of the NSCT coefficient to the texture feature, and it conforms to the Gaussian distribution. It is worth mentioning that since the filter of NSCT is nonsampled, a texture feature matrix equal in size to the original image can be obtained by the method; that is, each pixel in the image has a corresponding texture feature. This provides the basis for the construction of texture heterogeneity in the next subsection.

Multiscale segmentation incorporating edge feature

Algorithm flow

We propose an object-based remote sensing image MSS algorithm based on the edge merging cost criterion. The implementation process of the algorithm is shown in Fig. 4.

When computing the image object Obj1 and the image object Obj2 to Obj_Merge, the article’s algorithm defines the merging cost f generated during the merging as the weighted sum of the increase value of the object heterogeneity and the edge merging cost of the object: (19)${\displaystyle f={\omega }_{\text{color}}\cdot \Delta h_{\text{color}}+{\omega }_{\text{shape}}\cdot \Delta h_{\text{shape}}+{\omega }_{\text{texture}}\cdot \Delta h_{\text{texture}}+{\omega }_{\text{edge}}\cdot EdgeCost\left(Obj1,Obj2\right).}$

When performing bottom-up regional merging, the algorithm of the thesis adopts the criterion of minimum object merging cost as the regional merging strategy, combines the edge merging cost and the heterogeneity criterion to form a new merging criterion, and proposes a cost criterion based on the edge merging.

Evaluation analysis

Carleer, Debeir & Wolff (2005) evaluated the segmentation accuracy through two indicators: mis-segment ratio (MR) and regions ratio (RR). Figure 5 is a schematic diagram of wrong segmentation, where Fig. 5A is the reference segmentation result, Fig. 5B is the algorithmic segmentation result, and Fig. 5C is the algorithmic wrong segmentation map, in which the black area is the wrong segmentation part.

The lower the MR, the higher the overall accuracy of image segmentation, and vice versa. The calculation formula of the wrong segmentation rate is as follows: (20)${\displaystyle MR=\frac{\sum _{i=1}^{N_{\text{ref}}}\sum _{j=1}^{N_{\text{ref}}}{C}_{ij}-\sum _{k=1}^{N_{\text{ref}}}{C}_{kk}}{\sum _{i=1}^{N_{\text{ref}}}\sum _{j=1}^{N_{\text{ref}}}{C}_{ij}}\times 100\text{\%}.}$

C_ij represents the total number of pixels whose reference category is j divided into category i, and N_ref represents the number of reference result regions (i.e., the number of reference categories). The RR is used to comprehensively evaluate whether there is over-segmentation or under-segmentation in image segmentation. The RR is the ratio of the number N_s of the segmentation result area to the number N_ref of the reference result area. The closer the RR is to 1, the better the segmentation result. The calculation formula of the regions ratio is as follows: (21)${\displaystyle RR=\frac{N_s}{N_{ref}}.}$

Experiment 1

To verify the performance of the proposed method, we used three sets of data to test the algorithm. In the first experiment, five different textures in the Brodatz standard texture library were selected and synthesized into a texture image, as shown in Fig. 6A. In the second experiment, two typical textures in remote sensing images, architectural area and woodland, were selected and synthesized into a texture image, as shown in Fig. 6B. The remote sensing texture area is selected from the IKONOS satellite remote sensing image, and the image spatial resolution is 1 meter. The third experiment selected a part of the remote sensing image of the ZY-3 satellite in the Wuhan area with a spatial resolution of 2.1 m, as shown in Fig. 6C.

To analyze the performance of the proposed method and the MSS in segmenting images with rich texture information, we increased the weight of texture feature and reduced edge intensity. The parameters are manually tuned according to the degree of image texture and edge features. We chose the multiresolution segmentation algorithm in eCognition Developer 8.8 software (herein eCognition software) as the contrast experiments, and compared and analyzed the results of the three groups of experiments.

Figure 7 shows the segmentation result of the Brodatz texture synthetic image. Figures 7A and 7B show the segmentation results of eCognition software at different scales. The parameters in Fig. 7A are set to scale s = 30 and weight factor ω_shape = 0.9 ω_cmpct = 0.5. The parameters in Fig. 7B are set to scale s = 70 and weight factor ω_shape = 0.8 ω_cmpct = 0.5. Figure 7C is the segmentation result of the algorithm proposed in this article. The parameters are set to scale s = 100, weight factor ω_shape = 0.05, ω_texture = 0.95, ω_cmpct = 0.5. It can be seen from Fig. 7 that the algorithm proposed in this article divided five different texture types into five independent and complete regions. However, the segmentation results of the eCognition software failed to obtain a complete texture area at different scales, and there is a serious mis-segmentation at the boundary of the texture area.

Figure 8 shows the segmentation result of remote sensing texture synthetic image. Figures 8A and 8B are the segmentation results of eCognition software at different scales. The parameters in Fig. 8A are set to scale s = 50, weight factor ω_shape = 0.8, ω_cmpct = 0.5. The parameters in Fig. 8B are set to scale s = 80, weight factor ω_shape = 0.2, ω_cmpct = 0.5. Figure 8C is the segmentation result of the algorithm proposed in this article. The parameters are set to scale s = 50, weight factor ω_shape = 0.05, ω_texture = 0.95, ω_cmpct = 0.7. It can be seen from Fig. 8 that the algorithm proposed in this article can better distinguish the two different remote sensing textures of the construction area and the woodland and divided the image into four independent and complete areas. However, the segmentation results of the eCognition software failed to obtain a complete texture area at different scales, and there is obvious mis-segmentation at the boundary of the texture area.

Figure 9 shows the segmentation result of the resource No. 3 image in the Wuhan area. Figures 9A and 9B show the segmentation results of eCognition software at different scales. The parameters of Fig. 9A are set to scale s = 61, weight factor ω_shape = 0.7, ω_cmpct = 0.0. The parameters of Fig. 9B are set to scale s = 80, weight factor ω_shape = 0.3, ω_cmpct = 0.5. Figure 9C is the segmentation result of the algorithm proposed in this article. The parameters are set to scale s =150, weight factor ω_shape = 0.1, ω_texture = 0.8, ω_cmpct = 0.3. It can be seen from Fig. 9 that the algorithm proposed in this article can better distinguish the construction area in the central area of the image. The cultivated land distributed on the upper side, lower side, left side and right side of the building area is divided into four complete areas, and the building area is divided into two areas.

We adopted the method of supervision and evaluation, based on the error of the number of pixels, and evaluated the accuracy of the image segmentation results through two indicators: the MR and the RR. In Fig. 10, column (a) is the reference segmentation results of the three sets of experimental data, columns (b) and (c) are the mis-segmented diagrams of the eCognition software segmentation results of the three sets of experimental data. Column (d) is the mis-segmented map of the segmentation result of the algorithm proposed in this article, in which the white part is the mis-segmented pixel.

Table 1 evaluates the segmentation results of Experiment 1. It can be seen from Table 1 that the MR of the segmentation results of the proposed algorithm for the three groups of experiments in the article are lower than the segmentation results obtained using eCognition software.

Therefore, in the three sets of experiments, the eCognition software has different degrees of mis-segmentation at the boundary of the image texture area. In the later experiments, for the more complex texture area of the building area, the eCognition software has the phenomena of over-segmentation at different scales.

In summary, because the regions with rich texture information in the image show different spectral heterogeneity in the spectral feature, the traditional object-based MSS algorithm does not fully consider the texture information of the features in the image. Therefore, over-segmentation will occur when segmenting images with rich texture information.

Experiment 2

When segmenting images with less texture information and rich edge information, we increased the weight of the edge intensity and reduced the weight of the texture information. We selected a satellite remote sensing image of Milton Keynes, UK, as shown in Fig. 11A; the image was acquired in 2005. The area covered in the image includes the corner of a plant maze, and there are two dominant types of features, grass and roads. Through visual interpretation, we manually segmented the image and extracted eight roads and 10 grasslands separated by roads, totaling 18 areas. The reference segmentation result is shown in Fig. 11B. At the same time, we choose the MSS algorithm in eCognition software as a comparative experiment.

Figure 12 is the segmentation result of the first set of experiments. Figures 12A and 12B are the segmentation results by using the eCognition. The parameters of Fig. 12A are s = 70, ω_shape = 0.8, ω_cmpct = 0.5, while the parameters of Fig. 12B are s =150, ω_shape = 0.1, ω_cmpct = 0.5. Figure 12C is the segmentation result of this article’s algorithm, and the parameters are s =150, ω_color = 0.5, ω_edge = 0.5. Figures 12D, 12E and 12F indicates incorrect segmentation results of eCognition and our algorithm.

It can be seen from Fig. 12 that the algorithm proposed in the article divides the eight roads and 10 grasslands in the image into independent and complete objects. For the segmentation results of the eCognition software, when the shape weighting factor is large, the software segmented the road and grass in the image into several segments, failing to segment the complete objects, as shown in Fig. 12A. When the shape weighting factor is small, because the algorithm’s constraints on the shape feature are weakened, the road objects in the segmentation result have some burrs at different positions, as shown in Fig. 12B. The proposed algorithm made full use of the edge feature in the remote sensing image, and the location of the edge feature is more accurate.

Based on the error of the number of pixels, we evaluated the accuracy of the image segmentation results through two indicators: the MR and the RR. Table 2 shows the segmentation accuracy of Experiment 2 image by different algorithms. It can be seen from the table that the ratio of the segmentation results of eCognition software at different scales is greater than 1, and there are different degrees of over-segmentation. When the scale s = 150, the RR of the eCognition software segmentation result is closer to 1, but due to the lack of shape constraints, the positioning of the edge feature is inaccurate, and the MR is slightly higher than the result of the scale s = 70. The proposed algorithm divides the regions with similar features into independent and complete objects, the RR is equal to 1, and the MR is lower than the segmentation results of eCognition software at different scales.

Experiment 3

When segmenting images with rich texture information and edge information, we adaptively adjust the weights of textures and edges.

Figure 13A is the segmentation result of the algorithm proposed in Experiment 1, and Fig. 13B is the result of the object-based segmentation algorithm based on the edge merging cost criterion proposed in this section. The parameters in Fig. 13A are set to scale s =150 and weight factor ω_shape = 0.1, ω_cmpct = 0.3, ω_texture = 0.8. The parameters in Fig. 13B are set to scale s =150 and weight factor ω_shape = 0.1, ω_cmpct = 0.3, ω_texture = 0.7, ω_edge = 0.2. Figures 13C and 13D are the corresponding mis-segmentation diagrams, respectively.

It can be seen from Fig. 13 that after the algorithm incorporates the edge feature, the accuracy of the boundary location of the texture area has been significantly improved, and the upper and lower boundaries of the building area can be found more accurately, and the building is divided into an independent and complete area. Table 3 shows the segmentation accuracy of the two algorithms. It can be seen from the table that the segmentation result of the algorithm based on the edge merging cost criterion is lower in the error segmentation rate than the object-based segmentation algorithm combined with the texture feature, and the RR is equal to 1.

Through three sets of experiments, the article supports the effectiveness of the object-based MSS algorithm based on the edge merging cost criterion. While using the spectral, shape, and texture features of remote sensing images, the algorithm accounts for the edge feature of the image. The segmentation results can accurately locate the boundaries of the features, and its segmentation accuracy is better than traditional algorithms.