Soil organic carbon estimation using remote sensing data-driven machine learning

Study area

Located in the eastern coastal region of China, Shandong Province spans geographic coordinates from 32°18′28″N to 38°23′23″N and 114°19′50″E to 122°43′36″E, covering a total land area of approximately 157,900 km². Shandong Province exhibits a complex and diverse topography, including plains, mountains, hills, and coastal areas. The central part is characterized by high-altitude mountainous areas, while the western and northern regions consist of plain areas formed by the Yellow River’s alluvial deposits. Shandong Province falls within the temperate monsoon climate zone, primarily influenced by marine and monsoon climates, leading to notable variations in annual average temperature and precipitation levels among different regions (Fig. 1).

Shandong Province boasts a diverse range of agricultural land types, such as paddy fields, drylands, orchards, forests, and grasslands. The main soil types in Shandong Province include brown soil, cinnamon soil, tidal soil, saline soil, among others. Various land use types and conversions between farmland, forests, grasslands, buildings, and unused land significantly influence the accumulation and stability of SOC. Agricultural management practices, including fertilization, tillage, and irrigation, can impact the fluctuations in SOC content. Furthermore, socio-economic factors like fertilizer and pesticide usage, land ownership patterns, and rural economic development levels may affect the distribution and variability of SOC content.

Data resources

Screening of environmental factors

There are many environmental factors that affect SOC content, so it is important to screen the model variables before training the model. Since machine models are often referred to as “black boxes” that do not directly reveal the functional relationship between environmental factors and target variables, each variable needs to be eliminated one by one in order to determine its effect on the model. Variables were screened by increasing or decreasing the RMSE, retaining the variable when the RMSE increased, and excluding it when the RMSE increased.

Retrieval model of SOC

In this study, SVM, RF, and XGBoost machine learning models were selected to retravel SOC content. On the one hand, these machine learning models can effectively handle multidimensional datasets and Collinearity problem. On the other hand, these models were widely used in SOC retrieval and obtained good simulation effect (Hengl et al., 2017; Liu et al., 2022; Zhou et al., 2023). For example, SVM exhibits significant advantages in addressing small sample problems. It maintains robust classification performance even with limited sample sizes by constructing an optimal hyperplane to separate different data categories (Yao, 2022). RF enhances prediction accuracy and robustness by integrating multiple decision trees (Momade et al., 2020), while XGBoost supports various objective functions and evaluation metrics, enabling outstanding performance in diverse prediction tasks (Sagi & Rokach, 2021). Although studies indicate that deep neural networks (DNN) are more accurate than RF and SVM in SOC simulations in some areas, neural networks require extensive parameter settings and more sample data (Raczko & Zagajewski, 2017); given that a small sample size may not be optimal for this study, deep learning was not chosen.

In this study, SVM modeling was implemented using the “svm” function in the “e1071” package, with the “importance” parameter set to TRUE to calculate variable importance and the “proximity” parameter set to TRUE to calculate sample proximity.

RF modeling was implemented using the “randomForest” and “caret” packages in the R programming language. The default value of “ntree” was set to 500, and the “expand.grid()” function in the “caret” package is used, which determines the optimal value of the parameter mtry through grid search. Specifically, we performed a grid search on the mtry parameter, traversing a series of possible values and evaluating the impact of each value on model performance. The role of the expand.grid() function is to generate all possible parameter combinations for grid search.

XGBoost incorporates regularization techniques, efficient parallel processing, feature selection, and missing value handling, among other optimization techniques. The “xgboost” library was loaded using the “library(xgboost)” command. In this model, learning rate and depth of tree were gbtree, 0.4 and 7, respectively. In this study, a grid search strategy was used to fine-tune all parameters using the caret package in R software. The dataset was divided into a training set and a test set with a ratio of 7:3.

Model accuracy validation

In this study, the accuracy and stability of the models were evaluated using three commonly used metrics: coefficient of determination (R²), root mean square error (RMSE), and relative percent difference (RPD). Higher modeling accuracy indicates stronger stability and predictive capability of the model. A value of R² closer to 1 indicates a higher explained variance of the target variable, indicating better model fit. A lower RMSE value indicates smaller prediction errors and better model fit. As for RPD, its classification is detailed in Viscarra Rossel, Taylor & McBratney (2007).

(1) $$RMSE=\sqrt{{\displaystyle \frac{1}{n}{\sum }_{i=1}^n{\left(P_i-M_i\right)}^2}}$$

(2) $$R^2={\left({\displaystyle \frac{{\sum }_{i=1}^n\left(M_i-\overline{M}\right)\left(P_i-\overline{P}\right)}{\sqrt{{\sum }_{i=1}^n{\left(M_i-\overline{M}\right)}^2}\sqrt{{\sum }_{i=1}^n{\left(P_i-\overline{P}\right)}^2}}}\right)}^2$$

(3) $$\mathrm{R}\mathrm{P}\mathrm{D}={\displaystyle \frac{\mathrm{S}{\mathrm{D}}_{\mathrm{O}}}{\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}}.}$$

In the equation, P_i and M_i represent the simulated and measured values of SOC content (g/kg), respectively, while SDO represents the standard deviation of the observed values.

Statistical analysis of SOC content

The average SOC content of the soil samples in this study was 8.78 g/kg. According to the national soil nutrient classification standards, it can be classified as a moderate level (6–12 g/kg). The coefficient of variation (CV) of SOC content in the collected samples was 55.75%, which was a moderate level of variation (Table 2).

Meanwhile, SOC content of different vegetation was different as showed in Table 3. Forest had highest SOC content, mean value of SOC was 18.2 g/kg, but also has the maximum CV (88.7%). Reasons may be that the forest covered large area in Shandong Province, the elevation fluctuated greatly and the sampling site was sparse. SOC of Farmland and grassland was lower with a mean value of 8.5, and 8.1 g/kg, and has the moderate CV (belongs to 49.6–54.3%).

Accuracy comparison analysis

XGBoost demonstrates the highest predictive accuracy in the validation set, with an R² of 0.7548, RMSE of 7.6792, and RPD of 1.1311 (Table 4). This indicates that the model can account for 75.48% of the variance in the test data. XGBoost also exhibits the highest R² value of 0.9573 and RPD value of 3.1859 in the training set, suggesting exceptional predictive capability and accuracy. These findings are consistent with previous research results (Emadi et al., 2020; Nguyen et al., 2022; Ye et al., 2021). While all three models perform well on the training set, they show poor performance on the test set or new data. This highlights our limitation, as the insufficient training samples lead to overfitting. Due to the lower generalization error of the RF model, which is more robust to noise (Breiman, 2001) and capable of handling non-linear and hierarchical relationships between SOC and predictor variables (Zhang et al., 2017), some studies (Lamichhane, Kumar & Wilson, 2019) suggest that the RF model can better simulate SOC content. This contrasts with our study, which can be attributed to differences in research fields, feature variable selection, and spatial resolution choices.

Driving factors of SOC simulation analysis

RF was selected to analyze the explanatory power of each variable in predicting the surface SOC content. RF was chosen for feature variable analysis because it can handle a larger number of variables, improve the accuracy and performance of the classification task, efficiently identify correlated variables (Chen et al., 2020a), and generate p-values to determine significantly correlated features while controlling the false discovery rate (Paul & Dupont, 2015).

Figure 2 illustrates the significance of each independent variable in the analysis. The results show that elevation had the highest explanatory power at 21.74% in explaining SOC content. This suggests that altitude plays a crucial role in predicting SOC content, consistent with previous studies (Wang, 2021). Precipitation and temperature had explanatory powers of 11.12% and 5.58%, respectively, indicating a moderate effect on SOC prediction. Precipitation and temperature, as major climatic factors, influence SOC content and its spatial distribution, affecting crop growth and plant net primary productivity (Wang et al., 2018). SOC decomposition and accumulation are significantly influenced by climatic hydrothermal conditions. Climate warming accelerates the decomposition of SOC by microorganisms (Schuur et al., 2015). Regarding soil type, clay and silt soils exhibit higher explanatory power, amounting to 13.47% and 11.65%, respectively (Zinn, Lal & Resck, 2005). Nevertheless, clay content shows relatively weak explanatory power in predicting soil organic matter content, possibly due to the consideration of additional soil physicochemical factors (Rasmussen et al., 2018).

The explanatory power of other characterizing variables should also be considered. Population density demonstrates an explanatory power of 8.58%. Regarding the vegetation index, EVI had an explanatory power of 4.38%, while NDVI had 6.6%. In terms of microwave remote sensing, VH had an explanatory power of 7.64%, while VV had 5.05%. These results suggest that besides elevation, precipitation, temperature, soil type, and other environmental factors play significant roles in the spatial distribution of SOC content.

Spatial distribution analysis of SOC content

The spatial distribution mapping of SOC content was conducted based on the XGBoost simulation results (Fig. 3). To exclude regions with minimal soil coverage, an NDVI mask was applied to the land use classification image. The NDVI mask, established from literature (Yang, Di Girolamo & Mazzoni, 2007), used a threshold value of 0.2 for the NDVI image. Pixels with NDVI values below this threshold were considered to have low vegetation cover and were then masked out from the land use classification image. This process ensured effective exclusion of areas with limited soil presence, leading to a surface SOC content mapping that accurately represents soil-dominant regions.

The study results revealed that in the mountainous areas of central Shandong Province, the highest SOC content was observed in regions with high altitude and steep slopes. These regions, characterized by forest cover, exhibited higher SOC content, consistent with previous research (Guo et al., 2020), reaching up to 24.34 g/kg. This can be attributed to the dense vegetation cover, organic matter release by trees during growth (Zhang et al., 2019), favorable climatic conditions, and limited anthropogenic impacts, promoting organic matter accumulation and preservation (Wang, 2019; Wiesmeier et al., 2013). Furthermore, traditional land management practices, like terracing and agroforestry, passed down through generations, aid in soil erosion prevention and organic matter accumulation in the soil (Chen et al., 2020b; Wei et al., 2019). Cropland areas at low elevations in the western and northern parts of Shandong Province exhibit low SOC content. Common intensive agricultural practices in these areas, such as ploughing and fertilizer application, result in the loss and degradation of soil organic matter (Wang, Amundson & Niu, 2000). Adopting sustainable agricultural practices like conservation tillage and organic farming could potentially alleviate the decline in SOC levels in these areas and enhance long-term soil quality (Martínez-Mena et al., 2020). The economically developed coastal areas of Shandong Province exhibit lower soil organic matter content attributed to distinct environmental conditions and human activities like urbanization, industrialization, and agricultural development, resulting in land degradation and soil organic matter loss (Wang, 2019). Furthermore, the rise in sea levels can cause erosion and degradation of soil carbon reserves (Haywood et al., 2020). The proximity to the ocean can result in seawater intrusion, adversely impacting soil quality and the preservation of organic matter (Morrissey et al., 2014). Our results align with those of Dai et al. (2017), indicating that surface SOC density distribution follows a pattern of low levels in coastal areas, medium levels in the northwestern plains and eastern hills, and high levels in the mountainous regions of south-central Shandong Province.

Furthermore, in combination with Fig. 4, the study demonstrates variations in how different land use types impact SOC content. Land use in Shandong Province is predominantly cropland, with garden land and woodland following, while grassland, commercial and service land, industrial and mining land are less common. Significantly, woodland and garden land exhibited higher SOC content compared to cropland, aligning with previous studies (Edmondson et al., 2014; Fang et al., 2014, 2012) and clarifying the lower SOC content in Shandong Province. This phenomenon may be attributed to the over-utilization of cropland in Shandong Province (Liu et al., 2005) and extensive history of repeated ploughing, resulting in decreased SOC content. Optimal practices such as irrigation, fertilizer application, stubble return, and reduced tillage can enhance soil organic carbon storage and agricultural sustainability (Zhao et al., 2013).

The spatial distribution pattern reveals the dynamics of SOC accumulation and loss across various regions, offering crucial insights for formulating land management policies and executing land conservation strategies.