Enhancing geotechnical damage detection with deep learning: a convolutional neural network approach

Superficial erosion

Erosion occurs due to the wear of soil particles on the surface, entrapment of rainwater, wind, temperature fluctuations, and other geological agents, including gravitational forces (Borrelli et al., 2021). Soil erosion is a significant indicator of failure, which occurs when soil resistance drops due to various factors, including reduction of matrix suction stress, discontinuities, modification of the structure of sensitive soils, liquefaction of saturated sand, and loss of cohesion.

The erosive processes of the slopes are controlled by natural and anthropic factors, including rain erosivity, soil erodibility (vulnerability), the nature of the vegetation cover, vegetation cover, slope characteristics and types of soil use and occupation (Hudson, 1961).

Irregular terrain is more vulnerable to water erosion since splashing, surface runoff, and transportation all have more pronounced effects on steep slopes. Furthermore, soil characteristics and properties impact its vulnerability by affecting factors such as water infiltration rate, permeability, and water absorption capacity, as well as indicating the potential for dispersion, splash, abrasion, and transport caused by rainfall and flooding (Nguyen & Indraratna, 2020; Cooke & Doornkamp, 1990).

Vegetation cover reduces soil erosion rates because it protects against the impact of rain, decreases the amount of water available for surface runoff, decreases surface runoff speed, and increases the capacity of the soil for water infiltration (Cooke & Doornkamp, 1990). Therefore, vegetation cover is an important aspect when analysing vulnerable soil and its lack indicates possible failure.

Landslide

The term ‘landslide’ pertains to mass movements characterised by a distinct zone of weakness, separating the sliding material from the more stable underlying strata, as depicted in Fig. 1. This geological phenomenon is intricate, influenced by a multitude of factors, including slope geometry, soil quality, moisture content, precipitation, vegetation index, construction activities in the region, surface load, and proximity to roads and rivers (Terzaghi, Peck & Mesri, 1996; Aziz et al., 2021). The influence of these factors may vary between regions and geological and topographic conditions (Khan et al., 2021).

Landslides are classified by type of material and type of movement. Usually, the material is rock or soil. The two main types of slides are rotational slides and translational slides. The landslide phenomena can be a combination of two or more of the movement types (Varnes, 1978).

This article aims to investigate the application of computer vision, in combination with high-performance processing techniques such as CNN, to detect damage in geotechnical structures. The reliability of such structures is crucial for their effective functioning and the safety of the surrounding areas.

The following subtopics will provide the definitions and concepts necessary to understand the proposed solution effectively. These subtopics are arranged in the following order: deep learning for image classification, convolutional layer, activation layer ReLu, pooling layer, dropout, Adam optimiser and model performance assessment.

Convolution layer

The convolution layer is responsible for convolving the image patches to extract universal and increasingly complex features. The convolutional layer uses filters, Fig. 3, which perform convolution operations when scanning the input. These operations are performed on the entire image by sliding kernels and finding the dot product between the filter and the input image parts. Consider an input image I with dimensions of height H, width W, and C colour channels (e.g., RGB, where C = 3). Alongside this input, we have a set of K convolution filters, each with a height and width of (h × w) (also known as kernel size) and C = 3 channels to match the number of channels of the input image. The convolution operation on a single CNN layer can be precisely defined in Eqs. (1) and (2).

(1)${\displaystyle Z\left(x,y\right)=\sum _{c=1}^C\sum _{i=-\left(Fw-1\right)}^{Fw-1}\sum _{j=-\left(Fh-1\right)}^{Fh-1}I\left(x+i,y+j,c\right)\cdot K\left(i,j,c\right)}$ (2)${\displaystyle A\left(x,y\right)=f\left(Z\left(x,y\right)\right)}$

where Z(x, y) is the output of the convolution operation at a specific location (x, y) in the output feature map; c is the channel index, representing one of the C color channels in the input image and the corresponding filter channel; i is the horizontal offset of the filter relative to the current location (x, y) in the input image, and it ranges from −((Fw − 1)/2) to ((Fw − 1)/2), where Fw is the filter width; j is the vertical offset of the filter relative to the current location (x, y) in the input image, and it ranges from −((Fh − 1)/2) to ((Fh − 1)/2), where Fh is the filter height. I(x + i, y + j, c) is the value of the input pixel at position (x + i, y + j) in channel c; K(i, j, c) is the value of the filter at position (i, j) in channel c; and b_i is the bias term associated with filter c.

The output A(x, y) of this operation is named Feature Map, resulting from the application of the activation function to Z(x, y), which is the output of the convolution layer to the K filter. Which provides information about the corners, edges, or the features of the image and is read by other layers so that they can learn the remaining features of the image (Szegedy et al., 2015), such as illustrated in Fig. 4.

The convolution applied to image pixels automatically allows the identification and analysis of complex visual information. In contrast, support vector machines (SVMs), among other traditional techniques, typically require manual or semi-automated methods to extract features as an implementation of border filters, texture filters or histogram (Myagila & Kilavo, 2022, Chaudhari, 2018). Moreover, CNNs employ the convolution operation to mitigate their sensitivity to various image distortions and transformations, enhancing their performance in image recognition tasks.

Activation layer ReLU

Activation functions f(.) are commonly used after each convolution layer to insert a degree of linearity into the neuron output since the image data are not linearly separable (Wang et al., 2020). ReLU, represented by Eq. (3), is a widely used activation function considered among the most efficient for deep CNN. (3)${\displaystyle \text{ReLU}\left(x\right)=\left\{\begin{array}{cc}0,\hfill & \text{if}x<0\hfill \\ x,\hfill & \text{if}x\ge 0\hfill \end{array}\right.}$

Despite its appearance as a linear Function, Relu (as shown in Fig. 5) actually has a derivative function which enables backpropagation. However, when the input approaches zero or falls below it, the gradient of the function becomes zero, thereby hindering the ability of the network to perform backpropagation (Theodoridis, 2020; Limão, de Araújo & Frances, 2023). Unlike other nonlinear functions with bounded output values such as zeros, positives, and negatives, the Relu has no bounded outputs other than its negative input values (Cha, Choi & Büyüköztürk, 2017).

In practice, the outputs of the filters are submitted to the activation function at the end of each convolutional layer. After going through this process, they are used to calculate the errors and update the neural network weights.

Pooling layer

In general, images have a series of redundant information. Therefore, it is necessary to use the pooling layers inserted between several of the convolutional layers of the network to avoid substantial performance degradation (Akhtar & Ragavendran, 2020). It is a process over the convolution layer. Then, the pooling layer resizes the feature maps that resulted from the previous convolutions for the sake of dimensionality reduction. For the last step, these maps are transformed into vectors by the fully connected layer (Li et al., 2018).

The most suitable pooling technique for processing representations reliant on count statistics has consistently been identified as pooling. This selection is justified by its capacity to reduce the dimensionality of hidden layers by an integer multiplicative factor (Limão, de Araújo & Frances, 2023). This, in turn, enhances the performance in applications that entail the processing of a substantial number of images (Murray & Perronnin, 2014).

The Max Pooling operation involves sliding a window (often referred to as a kernel or filter) over the input image in a specified stride, and for each window position, selecting the maximum value within that window, as illustrated in Fig. 6. A pooling window (blue rectangle) is applied to a group of pixels in an image represented by filled circles. The red arrow indicates the maximum pixel within the pooling window.

The technique achieves position invariance over larger local regions and downsamples the input image by a factor of P along each axis. This is formally defined by Eq. (4). (4)${\displaystyle \text{Max\_Pooling}\left(A,x,y\right)=\underset{p,q}{max}A\left[x\cdot P+p,y\cdot P+q\right]}$

where A(x, y) represents the input feature map, x and y indicate the spatial coordinates within the feature map. The pooling window, with a size of P × P, is moved across the feature map. For each position (x, y), the max pooling operation selects the maximum value within the window from the feature map A. This process reduces the dimensions of the input feature map by a factor of P along each axis, resulting in the output pooling map. In this context, the stride S is set equal to the size of the pooling window P to ensure that there is no overlap between adjacent windows. Consequently, at each step, the pooling window moves P units horizontally and P units vertically. Leading to a faster convergence rate by selecting superior invariant features, which improves generalisation performance (Nagi et al., 2011).

Dropout

Due to complicated coadaptations, training a network with a large number of neurons frequently leads to overfitting. This problem occurs when a network successfully categorises a training data set but fails to generalise, leading to poor validation and testing data performance. Overfitting has frequently been a problem in the field of machine learning (Cha, Choi & Büyüköztürk, 2017).

Dropout layers are added to overcome this problem. The primary objective of dropout is to randomly disrupt connections between neurons in interconnected layers with a pre-defined dropout rate (Garbin, Zhu & Marques, 2020). Therefore, minimising these coadaptations allows a network to generalise training samples more effectively.

During training in a CNN layer, the dropout output (h_drop) is obtained by element-wise multiplication with a dropout mask r(x, y). This technique applies the dropout mask to the result of the max pooling operation, effectively deactivating a random subset of elements and reducing their activations, as represented by Eq. (5). (5)${\displaystyle h_{\text{drop}}\left(x,y\right)=r\left(x,y\right)\cdot \left(\text{Max\_Pooling}\left(A,x,y\right)\right).}$

The dropout mask is a matrix of the same size as the output feature map, where each element r(x, y) represents a randomly sampled value (0 or 1) drawn from a Bernoulli distribution. The probability of each element being 1 is denoted by d (dropout rate), which controls the overall proportion of neurons to be deactivated during training. (6)${\displaystyle h_{\text{drop}}\left(x,y\right)=\left\{\begin{array}{cc}r\left(x,y\right)\cdot \left(\text{Max\_Pooling}\left(A,x,y\right)\right)\hfill & \text{com probabilidade}1-d\hfill \\ 0\hfill & \text{com probabilidade}d.\hfill \end{array}\right.}$

During CNN layer inference, the output h_inference, Eq. (6), s obtained by scaling down the convolutional operation result of Max_Pooling with a factor of (1 − d) to account for dropout probability d and ensure consistency with the training phase. The dropout mask r(x, y) is not applied during inference, and instead, the scaling factor (1 − d) adjusts the activations.

Flatten and fully connected layers

The flattened layer converts the multi-dimensional feature maps from previous convolutional layers, including those processed by dropout, into a one-dimensional vector, as represented as Eq. (7). (7)${\displaystyle \text{Flatten}\left({\mathbf{h}}_{\text{drop}}\right)={\mathbf{X}}_{\text{flat}},}$

where h_drop is the input feature maps from convolutional layers of image I, and X_flat is the flattened output vector.

The flattened layer restores the input feature maps originating from the convolutional layers of an image into a linear vector, thereby facilitating further processing in fully connected layers.

The fully connected layer performs a crucial task in neural networks by conducting a matrix multiplication of its input with a weight matrix, which is then subjected to an activation function (Basha et al., 2020; Jayanthi & V. Murali Krishna, 2022). A single neuron in a fully connected layer can be expressed as Eq. (8). (8)${\displaystyle Y_i=f\left(\sum _{j=1}^N{a}_{ij}\cdot X_j+b_i\right),}$

The Y_i represents the output of neuron i in the fully connected layer. f is the activation function. a_ij is the weight that links neuron i to input j. X_j represents features from the flatten layer, where N is the total number of features coming from the previous layer. Finally, b_i is the bias for neuron i.

Adam optimiser

Adaptive optimisation enhances performance by seeking the optimal weights for a neural network, as its goal is to minimise the error function (the closer to zero, the better), leading to a reduction in the overall error of the network. The Adaptive Moment Estimation (Adam) algorithm is employed to adjust the weights of a model during the training process. So, it combines the best properties of the AdaGrad and RMSProp algorithms to provide an optimisation approach that handles noisy problems (Zhang, 2019).

Adam is recommended in machine learning to solve problems with heterogeneous gradients effectively. To achieve the direction and magnitude of the gradient, Adam exponential moving averages of gradients (first-moment) and exponential moving averages of the gradient squares (second-moment).

Consider the hyperparameters α is the Learning rate, β₁ is an exponential decay factor for the first-moment estimate, β₂ is an exponential decay factor for the second moment estimate, and ϵ a small constant to prevent division by zero. The Adam optimiser for a CNN can be described in four steps:

Step 1: Parameter initialisation

Initially, the first-moment estimate (m_t), second-moment estimate (v_t) are and the iteration (t) set to zero.

Step 2: Update of moment Eqs. (9) and (10) (9)${\displaystyle m_t={\beta }_1\cdot m_{t-1}+\left(1-{\beta }_1\right)\cdot g_t\text{and}}$ (10)${\displaystyle v_t={\beta }_2\cdot v_{t-1}+\left(1-{\beta }_2\right)\cdot \left(g_t^2\right)\text{where}{g}_t\text{is the current gradient.}}$

Step 3: Bias correction of moments Eqs. (11) and (12) (11)${\displaystyle m_t=\frac{m_t}{1-{\beta }_1^t}\text{and}}$ (12)${\displaystyle v_t=\frac{v_t}{1-{\beta }_2^t},\text{where}t\text{is the current iteration.}}$

Step 4: Weight update (13)${\displaystyle {\theta }_t={\theta }_{t-1}-\alpha \cdot \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ},\text{where}{\theta }_t\text{represents the parameters (weights) of the CNN.}}$

Therefore, the Adam algorithm principle is the update of the weights (Eq. 13) based on the corrected moments and moment estimates. It combines information about the direction and magnitude of the gradients to adjust the network weights adaptively. This process repeats for each iteration during CNN training so that weights can be adjusted accordingly to minimise the loss function and enhance the performance of the model.

Model performance assessment

When evaluating classification models, accuracy is a widely used metric to assess how well the model performs at predicting class labels for a given dataset (Ekundayo & Viriri, 2021; Adnan et al., 2022). Accuracy is presented as a per cent score between 0 and 100%, with 100% indicating that all predictions are correct and 0 meaning that none of the predictions is correct, as shown in Eq. (14). (14)${\displaystyle \text{Accuracy}=\frac{\text{Number of Correct Predictions}}{\text{Total Predictions}}.}$

Loss is assessed by assessing the disparity between the predictions of the model and the actual data labels. The primary purpose of the loss function is to measure the precision of the predictions of the model (Ekundayo & Viriri, 2021; Adnan et al., 2022). Minimising the loss throughout the training process is imperative by aligning the predictions of the model as closely as possible with the actual labels. One widely employed loss formula for binary classification problems is binary cross-entropy, which can be represented as Eq. (15). (15)${\displaystyle \text{Loss (Binary Cross-Entropy)}=-\frac{1}{N}\sum _{i=1}^N\left(y_{i}log\left(p\left(y_i\right)\right)+\left(1-y_i\right)log\left(1-p\left(y_i\right)\right)\right).}$

In Eq. (15) N is the total number of examples in the dataset, y_i is the true label, for example, i, and p(y_i) is the probability predicted by the model that example i belongs to the positive class.

The precision (P), recall (R), and F1-score metrics evaluate detection performance based on the results of predictions. Algorithm detection results can be categorised as True Positive (TP) for correct detection, False Positive (FP) for incorrect detection of nonexistent objects or misplaced detection, and False Negative (FN) for undetected ground-truth bounding boxes (Han et al., 2022; Adnan et al., 2022).

Precision, described by Eq. (16), is the ratio of true positives to all predicted positives made by a model. (16)${\displaystyle \text{Precision}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Positives}}=\frac{\text{True Positives}}{\text{all detections}}.}$

In Eq. (17), the recall measures the proportion of true positive predictions among all actual positive instances. (17)${\displaystyle \text{Recall}=\frac{\text{True Positives}}{\text{True Positives}+\text{False Negatives}}=\frac{\text{True Positives}}{\text{all ground truths}}.}$

The F1-score, Eq. (18) is a metric that balances Precision and Recall and is used for evaluating both simultaneously. (18)${\displaystyle \text{F1-Score}=\frac{2\cdot \text{Precision}\cdot \text{Recall}}{\text{Precision}+\text{Recall}}}$

An examination of recall information, precision, and F1-Score can yield valuable insights into the functionality of the model, particularly with regard to false positives, false negatives, and true positives. The confusion matrix is a powerful tool for visually representing how the model categorises samples into different categories (Azimi & Pekcan, 2020).

Furthermore, the ROC curve visually represents how a model predicts true labels in the binary classifier. The AUC metric measures the overall quality of the ROC curve and briefly summarises the ability of the model to distinguish between different classes at different classification thresholds (Azimi & Pekcan, 2020). The larger the area under the curve, the better the model performs. An AUC score of 1 means perfect classification, while a score less than 0.5 indicates random selection (Adnan et al., 2022; Cano et al., 2021; Azimi & Pekcan, 2020).

Augmentation

The image transformations can be based on movement concerning the axis of the image (geometric), changes in the shape of elements, size, and position within the image, or act on their pictorial composition (photometric). For example, flipping, rotating, cropping, shearing, rescaling and zooming are geometric DA, and noise injection, colour jittering, random erasing, and principal components analysis are photometric DA.

In this problem, the number of images in the data set of the train is duplicated (337 to 674 images). The geometric properties of flipping, rotating, and rescaling are applied only to the training dataset. In addition, we applied the pictorial composition transformation-based augmentation by a smoothing filter in the image. This process improves model generalisation ability when dealing with new data.

The objective of the data augmentation was to expand and diversify the images in the training set to improve the performance of the model (Ekundayo & Viriri, 2021). Figure 10 shows some examples of transformed images.

Sample division

A total of 674 images are randomly selected from the dataset to generate training, validation and testing sets. The pre-processed set with augmentation and training set images is fed into a CNN to build a binary classifier that can distinguish between ‘damage’ (negative output, probability close to 0) or intact (positive output, close to 1). The composition of the dataset includes training, validation, and test data, as shown in Table 2.