Efficient geospatial mapping of buildings, woodlands, water and roads from aerial imagery using deep learning

Introduction

Monitoring and assessing land cover and land use are key components of natural resource management. In many sectors, such as urban planning, vegetation monitoring, military reconnaissance, and biodiversity monitoring, comprehensive information about the land cover or land use is a significant resource (Zhou, Huang & Cadenasso, 2011; Pauleit & Duhme, 2000; Ahmed et al., 2017; Gerard et al., 2010). Urban and rural areas have relied heavily on land descriptions provided by satellite data and image processing tools to spot changes. Environment change, woodland dynamics, and destruction can all be inferred from shifts in the landscape (Weber & Hall, 2001; Wickham, O’Neill & Jones, 2000; Potapov et al., 2015; Kennedy, Yang & Cohen, 2010). The rate of urbanization, agricultural intensity, and other human-made changes can be investigated by analyzing landscape dynamics such as fluxes between different land covers (Feranec et al., 2010).

Most studies in this area rely on multispectral aerial imagery, which can be helpful for various applications. Free satellite data normally has a resolution of 10–30 m, but commercial aerial images with a higher resolution can be expensive (Wulder et al., 2012). On the other hand, aerial images are often obtained by local and state governments and can offer pixel sizes as small as 25–50 cm or even smaller. Using aerial imagery to assess land parcel content properly is highly economically important for agricultural and public administration. This is because the government allocates resources and collects taxes based on the intended purpose of the land, be it for agriculture or construction. However, frequent changes in parcel appearance and usage due to human activity require the acquisition of new aerial orthophotos to identify any changes that have occurred. In this regard, classifying and vectorizing all emerging or disappearing objects is necessary. This includes the need for government entities that dole out subsidies to update lists of items that no longer qualify for those incentives. In economics, the most common things that change over time are structures like buildings and trees like forests, rivers, and highways. The usual practice for classification in land cover involves manual methods or simple image classification techniques, such as an object or pixel-based classification (Blaschke, 2010; Khatami, Mountrakis & Stehman, 2016; Caccetta et al., 2016; Ahmed et al., 2017).

Operators examine each orthophoto image in the manual method and mark the “new” objects by hand. This process can take months for the entire country and is highly prone to errors. Our experience indicates an average error rate of over 30% for multiple years over more than 312,000 square km. As a result, manually mapping large land areas is costly (Gerard et al., 2010). As a result, there is a pressing need to create an effective automatic tool to speed up processing while maintaining high precision. A variety of GIS-based software programs offer such tools. The conventional approach in computer vision, which involves manual feature extraction and rule-based methods, must be revised when dealing with high variance and large-scale data. This leads to significant effort and poor scalability. However, because of their scalability, affordability, and high performance, deep learning approaches employing convolutional neural networks (CNN) have become more relevant for automatic classification of aerial images (Khan et al., 2018; Maggiori et al., 2016; Vargas-Munoz et al., 2017). By analyzing the semantic division of certain areas at different periods, CNN enables the rapid and accurate monitoring of considerably wider regions and the classification over time. Unfortunately, big datasets with ground truth annotations are generally necessary for the deep learning strategy. Although aerial images are readily available, the use of computer vision for land cover segmentation is constrained by the difficulty and expense of collecting high-quality annotated datasets. Recently released datasets include some with detailed annotations, while others rely on less precise labeling in training data to identify things like buildings and roads. However, these data sets require methodically segmenting urban areas, forests, bodies of water, and roadways. We present LandCover.ai, a dataset optimized for semantic segmentation, which addresses this shortcoming by including those above four manually annotated classes. In the middle-sized European nation of Poland, we captured images of 216.27 square km of land at 50 cm per pixel and 176.76 square km at 25 cm per pixel. We also supply a model’s baseline findings for use in making judgments. This project aims to develop an automated, highly accurate method of aerial imagery analysis to provide data insights on land cover. This research uses semantic segmentation in computer vision to show how to use deep learning to obtain such insights. This article makes the following contributions.

• This research contributes to developing an automated system for analyzing aerial images to extract useful information about the land cover.

• Augments the data by generating new image samples during training to improve the effectiveness of the proposed model.

• Outperformed state-of-the-art research studies and other DL models in a comprehensive evaluation.

The following is the planned framework for studying the categorization of the LandCover.ai dataset: We review the literature on land cover uses classification in “Literature Review” and give an overview of the dataset in “Dataset and Preliminaries”. In “Proposed Approach”, this research can observe that DL is also considered for categorization purposes in this work. In “Experimental Analysis and Results”, we present the findings of this study. “Conclusion” discusses what these results mean and what directions future studies could go.

Literature review

In remote sensing and geospatial analysis, land cover mapping is essential because of the wealth of information it yields for uses, including urban planning, natural resource management, and environmental monitoring. Manual interpretation of aerial imagery is commonly used in conventional land cover mapping approaches, which are both labor-intensive and expensive. Deep learning techniques have greatly aided automatic land cover mapping from aerial imagery in recent years.

For land cover mapping, some studies have turned to deep learning methods. A deep CNN was developed, for instance, by Zhang et al. (2020), to categorize high-resolution remote sensing images. They classified images down to the pixel level using a fully convolutional network (FCN) architecture. They proposed it as a highly efficient, computationally effective method. With the suggested method, high-level features such as spectral patterns, geometric traits, and contextual information are fused to perform per-object categorization. The CRF is then used to fine-tune the per-object categorization result using contextual advice at the pixel level. Five standard approaches, including the current gold standard network DeepLabV3 +, are used to evaluate the approach. Image processing experiments on various datasets show that the previous technique performs exceptionally well for VHRI classification, especially on small datasets. It also demonstrates excellent computing efficiency during the model training and inference processes.

Deep CNNs offer a significant advantage over previous manual methods in speed; however, they require appropriately annotated data. Aerial or satellite photography requires a different perspective and a fine collection of classes than typically used in segmentation datasets developed for common items or street views (Cordts et al., 2016; Lin et al., 2014). Aerial and aerial images are present in some datasets, although these images are often formatted for image classification rather than segmentation. One such example is the UC-Merced dataset (Yang & Newsam, 2010), which was created to classify whole images and contains 21 categories, such as buildings, forests, and rivers, but its resolution of 30 cm per pixel needs to be improved for segmentation tasks. Similarly, Google Maps is the primary source for the image classification datasets (WHU-RS-RSSCN7-AID-NWPU_RESISC45-PatternNet) (Laban et al., 2018; Boguszewski et al., 2021; Xia et al., 2017; Zhou et al., 2018). In order to achieve their potential, deep convolutional neural networks require well-annotated data, as was discussed above. Aerial or satellite photography datasets require a different perspective and an adequate number of classes than those used in traditional segmentation datasets, which often focus on common items or street views. Most aerial and satellite imagery datasets were developed for picture classification. For instance, UC-Merced, one of the earliest satellite datasets, has to be updated for the segmentation task because it is meant for whole-image classification and has a resolution of 30 cm per pixel and 21 categories. There are additional Google Maps picture classification datasets that are very comparable.

However, datasets like DOTA and iSAID are made for multi-class detection and instance segmentation, respectively (Azimi et al., 2019; Waqas Zamir et al., 2019). These datasets have many categories, such as vehicles, bridges, and ships, but they need to be improved for natural resources management, which is a responsibility of public agencies. However, while helpful, datasets developed specifically for building or vegetation segmentation must be more comprehensive for land cover change detection. In addition to Agriculture-Vision, a large aerial image database for agricultural pattern analysis, there are many other databases available, such as the 2010 Tree Cover dataset for the Metropolitan Region of Sao Paulo, the Massachusetts Buildings Dataset, the Inria Aerial Image Labelling Dataset, the AIRS Automatic Mapping of Buildings Dataset, and many more.

Some datasets are based on unmanned aerial vehicles (UAVs), such as ERA, UCLA Aerial Eve, and Okutama-Action; they are designed for event recognition rather than land cover change detection (Mou et al., 2020; Barekatain et al., 2017). The ISPRS Vaihingen and Potsdam datasets are mentioned as manually annotated but are small and have limited coverage of urban areas. Additionally, they need the water class. On the other hand, the Chesapeake Bay Land Cover Dataset (Ortiz et al., 2020) covers a larger area but has a lower resolution and is automatically annotated. While it has many useful classes, it does not include the “building” category. In summary, although several datasets are available for aerial imagery, there is a need for more comprehensive datasets with a broader range of classes, higher resolution, and accurate manual annotation. High-resolution remote sensing footage of both urban and rural areas has been collected, and a new technique is proposed in this article to extract roads from this data (Gao et al., 2019). The technique has two main components: RDRCNN and a TV algorithm for post-processing. Dilated convolution operators form the basis of the RDRCNN model, which is based on ResNet and U-Net architectures. The RDRCNN outputs are combined with the roadways’ line salient features to rejoin the separate areas. The main contributions of this research are the improved CNN architecture for road region detection and the blind voting method for merging regions with misclassification due to visual occlusion. Two difficult datasets are used to test the approach, one created by our researchers. We also compare this strategy to others. While the experimental results show that the method is effective at road extraction from complicated backgrounds (both urban and rural), further processing is needed to draw boundaries more precisely, especially in urban regions.

Wang et al. (2023) proposed DeepDSFusion, a unique decision-level fusion approach based on the Dempster-Shafer (DS) theory. This approach is generally compatible with any other pixel-level segmentation model. Several conventional data augmentation techniques were used in the DeepDSFusion architecture detail to get multiscale probability maps, including rotation and scale transformations. Then, using the DS theory, multiscale probability maps were fused into a single probability map. In order to get segmentation results, a straightforward threshold is finally used in the single probability map. The efficacy of DeepDSFusion is demonstrated by three traditional pixel-level segmentation tasks using high-resolution imagery: landcover mapping, road extraction, and deforestation identification. In Amritesh et al. (2023), the authors thoroughly analyze the outcomes of UNet and QGIS for land-use categorization at higher zoom levels using CNN approaches like UNet. They provide a UNet approach that works remarkably well in Indian circumstances. The solution is excellent for zoom level for cities and neighborhoods inside a city, which is very helpful for appropriate planning. In order to recognize roads and gather information on the different kinds of buildings in the region, they have also connected the UNet model with QGIS. These integrations will help their model identify and anticipate prospective growth areas. The author proposes dividing the landscape retrieval work into smaller projects to pinpoint distinct ideas present in satellite photos (Karatsiolis, Padubidri & Kamilaris, 2023). To detect these ideas, their method uses many models trained on Google Earth photos using unsupervised representation learning. They demonstrate the effectiveness of matching individual ideas to retrieve a scene or landscapes that resemble a satellite image of the Republic of Cyprus’s geographical area that the user has picked. Their findings highlight the advantages of segmenting the landscape similarity problem into discrete ideas strongly associated with remote sensing instead of using a single model that aims to target all underlying concepts.

Compared to earlier works in this section, the proposed approach contributes to developing an automated system for analyzing aerial images to extract meaningful land cover information using data-augmented and deep-learning approaches.

Dataset and preliminaries

We utilize a dataset that is both extensive and diverse, with manual annotation based only on three color channels (RGB) data, to allow for effective training of a model for precise semantic segmentation. For this purpose, this study used the LandCover.ai dataset (Boguszewski et al., 2021). The dataset images are categorized into four classes based on their relevance and importance for public administration purposes: building (1), woodland (2), water (3), and road (4). Buildings are considered objects that remain in a fixed location, excluding greenhouses. Each structure is only tagged for its roof and visible walls because our images are not full orthophotos. In contrast to single trees and orchards, woodland refers to an area where several trees grow in proximity. Water comprises both moving and still water, such as rivers and lakes, but not dry riverbeds or ditches. The road includes road and rail transport infrastructure, including parking lots and unpaved roads. Finally, background refers to areas not classified under any of the above classes, such as fields, grass, pavements, and other objects not included in the classes mentioned above. The different land used in the dataset is mentioned in Table 1. The purpose of Table 1 is to provide a clear and organized summary of the different land covers in the dataset, their corresponding area covered, and their assigned numerical label. The Class column lists the land cover type, such as Background, Building, Woodland, Water, and Road. Our study utilized label datasets with dimensions of 512 × 512 pixels. Each pixel in these label datasets represents one of five classes considered in our analysis. These classes were meticulously defined based on the specific criteria relevant to our study. The label datasets served as the ground truth for evaluating the performance of our proposed model, allowing for precise classification and segmentation tasks. The Coverage (km) column indicates the area covered by each type of land cover in kilometers. The Label column assigns a numerical label to each type of land cover.

The dataset comprises photographs used to generate an aerial digital orthophoto of Poland. The reference data of the LPIS is continually updated using images collected from the public geodetic resource. Cartesian “1992” (EPSG:2180) coordinates were used to create the digital orthophotos. The images’ spatial resolution was either 25 or 50 cm per pixel, and they were taken using three spectral bands (RGB). These images were taken between 2015 and 2018, and the average flying season in Poland is between April and September. As a result, the obtained images cover various optical conditions at various times of the year and stages of the vegetative cycle. This data set is massive and versatile (Boguszewski et al., 2021).

To ensure maximum diversity, we carefully handpicked 41 orthophoto tiles from different counties across all regions of Poland, as illustrated in Fig. 1. Each tile covers an area of approximately 5 km². Out of the 41 tiles, 33 have a resolution of 25 cm (roughly 9,000 × 9,500 pixels), while the remaining eight have a resolution of 50 cm (approximately 4,200 × 4,700 pixels), resulting in a total coverage of 176.76 and 39.51 km² respectively, amounting to a total area of 216.27 km². Figure 2 shows a selection of sample images.

Proposed approach

The proposed methodology of this work is presented in Fig. 3. The proposed methodology for the LandCover.ai dataset classification involves several key steps. Firstly, the dataset images and their corresponding masks were divided into 512 × 512 tiles, and smaller ones on the right and bottom edges were discarded. The tiles were then randomly shuffled and organized into the train, validation, and test sets. To further enhance the training set, offline augmentation was applied using imgaug, resulting in a total of 74,700 samples in the training set. The proposed methodology also includes using three different models for the segmentation task: the Vanilla UNet model, the UNet model with the Resnet50 encoder, and the DeepLabV3 model with the Resnet50 encoder. These models were defined using the PyTorch-based framework, SMP library, with varying parameters such as encoder name, encoder weights, decoder atrous rates, and encoder output stride. The proposed approach utilized accuracy, precision, recall, F1-score, model training and validation loss, and model predicted score to evaluate the performance of the deep learning models used in this work. The proposed methodology aims to improve the detection accuracy and robustness of the LandCover.ai dataset classification through appropriate data preprocessing and model selection.

Experimental analysis and results

The experimental results of the proposed approach are explained in this section. The study used the LandCover.ai dataset for the experiments. Three sections of the dataset were separated. The training, validation, and test sets were divided at a percentage of 70%, 15%, and 15% respectively. The training set’s pixel distribution for each class for 512 × 512 pictures. The experiments were performed using Google Colab due to its open-access GPU computation. The study used three deep learning models (Vanilla-UNet, ResNet50 UNet and DeepLabV3 ResNet50).

Table 2 shows the experimental results of three models on a given dataset: Vanilla-UNet, ResNet50 UNet, and DeepLabV3 ResNet50. These models are evaluated using performance metrics such as accuracy, precision, recall, and F1-score. The three models completed the training using the 28, 20 and 30 epochs.

The Vanilla-UNet model achieves an accuracy of 91.31% with a recall of 92.10%, a precision of 93.10%, and an F1-score of 92.50%. Training and validation loss for the Vanilla-UNet model is depicted in Fig. 4. Training lasts for 28 iterations. The initial values for the training and validation losses are 0.6967 and 0.6628, respectively. The training and validation losses reduce as the model is trained for more iterations. Each epoch includes a report detailing the training and validation loss, with the best model having the lowest loss. An anti-overfitting method, EarlyStopping counts the number of epochs during which the validation loss has not decreased before stopping training. The training above uses an early ending criterion of 4, indicating that the training will be terminated if the validation loss does not decrease for four epochs in a row. At epoch 28, the early stopping criterion is met, and the training is stopped. The final training loss is 0.3141, and the final validation loss is 0.2961.

ResNet50 UNet outperforms Vanilla-UNet with an accuracy of 94.37% and a precision of 94.80%. However, it has a lower recall of 94.20% and an F1-score of 93.70%. Figure 4B shows the training and validation loss of the ResNet50 UNet model. In the first epoch, the training loss was 0.5436, and the validation loss was 0.3822. The validation loss decreased compared to the initial value, and the model was saved. In subsequent epochs, the training loss decreased, indicating that the model improved on the training data. The validation loss also decreased, indicating that the model generalized well to the validation data. The model was saved every time the validation loss decreased. In epoch 8, the validation loss increased compared to the previous epoch, which triggered early stopping (a technique to stop the training process early if the model is not improving anymore). The early stopping counter was set to 5, which means that if the validation loss did not decrease for five consecutive epochs, the training would stop. In epochs 12 and 15, the validation loss decreased compared to the previous epochs, and the model was saved again. However, after epoch 15, the validation loss stopped decreasing, triggering early stopping after five consecutive epochs. The final validation loss was 0.2241, which was achieved in epoch 20.

DeepLabV3 ResNet50 has the highest accuracy of 94.77% with the highest precision, recall, and F1-score of 95.20%, 95.60%, and 95.40%, respectively. Figure 4C shows the training and validation loss of the DeepLabV3 ResNet50 model. The validation loss decreases from 0.365216 to 0.218037 over 24 epochs, indicating that the model’s performance improved significantly during this period. The training loss also decreased from 0.5107 to 0.1951 during the same period. The training and validation losses then start to fluctuate around 0.2–0.3, and the early stopping criterion is met after 29 epochs because the validation loss does not improve further. However, the training and validation losses provide insight into the model’s performance during the training process and can be used to optimize the model’s hyperparameters and prevent overfitting.

A broader comparison of these models indicates that DeepLabV3 ResNet50 performs better than the other models in accuracy, precision, recall, and F1-score. However, ResNet50 UNet also performs well with relatively high precision, while Vanilla-UNet has the lowest performance among the three models. These results suggest that using a more sophisticated architecture, such as ResNet50 and DeepLabV3, can improve the segmentation performance, which can be useful for various applications such as object detection and medical imaging.

Conclusion

With the increase in the availability of satellite imagery, enormous amounts of data are being collected daily. This spatial imagery provides valuable information that various entities can use. However, manually analyzing each image to obtain insights into land cover takes time and effort. To address this issue, this study proposes using deep learning techniques to accurately and efficiently classify satellite imagery pixel by pixel for land cover analysis. This study used the LandCover.ai dataset for model experiments. The three most important deep learning models (Vanilla-UNet, ResNet50 UNet and DeepLabV3 ResNet50) are used in this study. The proposed models show better performance compared to the existing approach. The Vanilla-UNet model has an accuracy of 91.31%, ResNet50 UNet obtained 94.37% accuracy, and DeepLabV3 ResNet50 has 94.77% accuracy. DeepLabV3 ResNet50 performs better than the other two models regarding accuracy, precision, recall, and F1-score. In the future, we intend to collect more data on land cover and apply semantic segmentation.

Supplemental Information