Innovative road distress detection (IR-DD): an efficient and scalable deep learning approach

The IR-DD model system

The IR-DD deep learning approach leverages the YOLOv8 detection model, renowned for its swift and accurate object detection capabilities, all achieved without the necessity of a regional proposal network. This system undergoes optimization to streamline the parameter count required for detection, enhancing its overall efficiency. Employing computer vision techniques, the IR-DD deep learning approach autonomously identifies road distress anomalies within images and video streams. Data preprocessing and object detection are the essential phases of the IR-DD model.

Object detection

Object detection involved several stages like model selection, training, evaluation, deployment, integration and maintenance which is illustrated in Fig. 3. In object detection phase, we initiate the model development process by thoughtfully selecting an object detection algorithm for training. We evaluate various options, including Faster R-CNN, SSD and YOLOv8, each having its unique set of advantages and limitations. However, we have selected YOLOv8 as the algorithm for our model. YOLOv8 is highly esteemed for its capability to achieve a harmonious balance between speed and accuracy, aligning seamlessly with the particular requirements of our system. Following the training phase, we evaluate the model’s performance using crucial metrics such as accuracy, precision, recall, and F1 score. In the event that the model does not meet expectations, we implement fine-tuning by adjusting hyperparameters or enhancing the dataset. This iterative process aims to strike a balance between overfitting and underfitting, ultimately refining the model’s accuracy and reliability.

Deployment is a pivotal phase that entails integrating the trained model into a real-time system for processing live video streams from road cameras. This process demands significant computing power and GPU support to facilitate real-time analysis. The system identifies road distress anomalies and utilizes a confidence threshold to manage false positives, ensuring precise and reliable detection. Integration is another pivotal stage where we seamlessly merge the road distress detection system with complementary systems like emergency response, traffic management, and maintenance alerts. This integration enables rapid responses, including road closures and maintenance alerts, ultimately optimizing road safety and response efficiency. Maintenance plays an ongoing role in upholding the effectiveness of the deployed system. It involves continuous model enhancement, routine testing, and meticulous upkeep of hardware and software components. These efforts ensure the system’s ongoing reliability and adaptability to changing conditions, all contributing to the promotion of road safety.

In the IR-DD model, our approach utilizes computer vision to swiftly identify road distress anomalies in real-time, whether from live camera feeds or pre-recorded video and image files. As depicted in Algorithm 1, it leverages a pre-trained YOLOv8 object detection model, trained on a diverse dataset that includes both road distress and non-distress images.The model functions by taking a dataset of video/image frames as input and generating outputs that identify anomalies, covering diverse road distress classes such as cracks, potholes, or surface damage. This model systematically processes each frame within the video stream. Before forwarding frames to the YOLOv8 model for object detection, the system initiates image pre-processing techniques customized for the current frame. If the model identifies any road distress anomaly, it promptly triggers an alarm and notifies the relevant authorities without delay. Furthermore, it archives the output video/image, duly emphasizing the detected road distress anomalies. The IRDD approach stands as a robust solution for real-time road distress detection, fostering swift and effective responses to potential road hazards.

Algorithm 1:

IR-DD algorithm.

Input: Dataset

Output: Detected Objects

Steps:

1: Initiate the input source for video/image (pre-recorded video/image file or live camera).

2: Start the video/image capture process.

3: while there are frames in the video/image do

4:  Implement image pre-processing techniques on the current frame.

5:  Send the pre-processed frame to the YOLOv8 model for object detection

6:  Examine the identified objects for classes related to road distress, such as cracks, potholes, and surface damage.

7:  if A class associated with road distress is identified then

8:   Trigger an alarm.

9:   Notify the relevant authorities.

10:  end if

11:  Save the resulting video/image, emphasizing the identified road distress anomalies.

12: end while

13: Halt the process of capturing video or image.

DOI: 10.7717/peerj-cs.2038/table-4

The IR-DD model algorithm

The IR-DD algorithm, driven by YOLOv8, integrates cutting-edge technology with systematic workflow to enhance road safety by rapidly identifying and reporting road distress anomalies. Its systematic approach ensures that distress situations are identified, addressed, and recorded efficiently, making it a valuable tool in the maintenance and management of road infrastructure.

The IR-DD algorithm, leveraging the high-performance YOLOv8 object detection model, unfolds through a meticulously orchestrated sequence of stages, each contributing to the overall efficacy of the system.

• Defining the video/image source: The initial stage involves the specification of the video/image source, whether it be a live camera feed capturing real-time data or a pre-recorded video/image file. This crucial step ensures that the system is appropriately aligned with the intended application scenario.

• Video/image frame capture: With the video/image source established, the system commences the video/image capture process. Frame by frame, it systematically processes each segment of the video/image, ensuring that no pertinent information is overlooked.

• Image pre-processing: Prior to subjecting each video/image frame to object detection, the system employs a suite of image pre-processing techniques. These techniques aim to enhance the quality and utility of the input data, which is essential for accurate and robust detection. The pre-processed frame is then forwarded to the YOLOv8 model for comprehensive object detection.

• Object classification: The YOLOv8 model plays a pivotal role in this stage. It rigorously examines each frame, categorizing detected objects into various classes. In the context of road distress detection, these classes typically include cracks, potholes, and surface damage. This classification process is a critical component of the system, as it enables the precise identification of distress-related anomalies.

• Alert triggering: The system is designed to respond promptly and decisively. In the event of detecting an object belonging to a road distress-related class, such as a crack or pothole, the system triggers an alarm. This alarm serves as a real-time notification mechanism, promptly relaying the information to the relevant authorities. This immediate response ensures swift and targeted action to address road anomalies and enhance safety.

• Video/image capture conclusion: As the video processing nears its conclusion, the system meticulously conserves the output video/image. In this recorded video/image, the detected road distress anomalies are distinctly highlighted. This visual representation is invaluable for post-event analysis, road maintenance planning, and record-keeping, offering a comprehensive record of the distress occurrences.

Bidirectional feature pyramid network

The incorporation of Bidirectional Feature Pyramid Network (BiFPN) into the YOLOv8 model for road distress detection significantly elevates the model’s capacity to capture multi-scale features, thereby enhancing its efficacy in detecting road anomalies. In the realm of road maintenance and safety, the accurate identification of distress anomalies such as cracks and potholes is pivotal for ensuring road quality and driver safety. The integration of BiFPN into the YOLOv8 framework equips the model with the ability to efficiently analyze features at different scales, enabling precise detection of distress anomalies across various road conditions and sizes. BiFPN’s bidirectional connections within the feature pyramid network facilitate seamless information flow both upward and downward through the network layers. This bidirectional flow enhances the model’s contextual understanding of road scenes, allowing it to discern subtle distress patterns and variations in road surfaces. The synergistic integration of YOLOv8 and BiFPN empowers the model to dynamically adapt to diverse road environments, ensuring robust performance in detecting anomalies irrespective of scale or complexity.

Furthermore, the integration of BiFPN not only enhances the accuracy of distress detection but also improves the efficiency of real-time anomaly identification. This advancement in intelligent transportation systems holds great promise for enhancing road maintenance practices, optimizing safety measures, and ultimately contributing to smoother and safer driving experiences for all road users. Algorithm 2 illustrates the YOLOv8 model enabled with BiFPN.

Mathematically, the BiFPN process, when integrated into YOLOv8 for road distress detection, can be represented as follows:

(1) $$P_i^{\mathrm{u}\mathrm{p}}=P_{i-1}+\mathrm{U}\mathrm{p}\mathrm{S}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}(P_i^{\mathrm{d}\mathrm{o}\mathrm{w}\mathrm{n}})$$

(2) $$P_i^{\mathrm{o}\mathrm{u}\mathrm{t}}=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}(P_i^{\mathrm{u}\mathrm{p}}))$$

Here, $P_i^{\mathrm{u}\mathrm{p}}$ represents the upsampled feature map from the lower-level pyramid, $P_i^{\mathrm{d}\mathrm{o}\mathrm{w}\mathrm{n}}$ is the feature map from the higher-level pyramid, and $P_i^{\mathrm{o}\mathrm{u}\mathrm{t}}$ denotes the fused feature map. This fusion process, when applied within the YOLOv8 model, ensures that distress anomalies on the road are detected accurately and reliably, contributing to improved road safety and maintenance efforts. The YOLOv8-BiFPN model, optimized for Road Distress Detection, can efficiently identify and localize road anomalies, providing an invaluable tool for infrastructure management and maintenance.

Following steps are follow for integrating BiFPN into YOLOv8 model for road distress detection:

• Data preparation: Prepare a dataset of road distress images with annotations (e.g., bounding boxes for distress anomalies).

• Backbone feature extraction: Utilize a pre-trained or custom backbone (EffcientNet) to extract features from the input images.

• BiFPN integration: Incorporate the BiFPN module into the YOLOv8 architecture after the backbone network. Configure the BiFPN to fuse features from different pyramid levels bidirectionally.

• Feature fusion: Implement feature fusion within the BiFPN to capture multi-scale features efficiently.

• Object detection head: Follow the BiFPN with an object detection head. Design the head to predict bounding boxes, object confidence scores, and class probabilities.

• Training: Train the YOLOv8-BiFPN model on the road distress dataset, using ground truth annotations. Use loss functions to optimize the model.

• Inference: Apply the trained YOLOv8-BiFPN model for road distress detection on new images.

• Post-processing: Implement post-processing steps like non-maximum suppression to filter and refine the detection results.

• Evaluation: Assess the model’s performance by utilizing metrics such as precision, recall, F1-score, and mean average precision (mAP).

• Fine-tuning and optimization: Optionally, fine-tune hyperparameters and model architecture to achieve optimal performance.

Anchor-free detection

This section provides a comprehensive exploration of anchor-free model architecture, elucidating its significance in the realm of object detection tasks. Understanding how the anchor-free model is structured and carefully adjusting our model’s settings is a key strategy to improve the performance of our object detection model, especially when dealing with objects that have irregular shapes. YOLOv8 adopts an anchor-free model architecture, deviating from the conventional approach of predicting offsets from predefined anchor boxes. Instead, it directly predicts the center of an object, eliminating the reliance on anchor boxes for bounding box predictions. To visually represent the concept of anchor boxes in object detection, Fig. 4 illustrates the graphical depiction of anchor boxes (Solawetz & Francesco, 2023). These boxes are pre-defined bounding boxes with varying scales and aspect ratios, serving as crucial elements in object detection algorithms for anchoring and predicting objects within an image. The network generates predictions for five bounding boxes at each cell of the output feature map. For each bounding box, the network predicts five coordinates: $t_x$, $t_y$, $t_w$, $t_h$, and $t_o$. If the cell is displaced from the top-left corner of the image by the offset ( $c_x$, $c_y$), and the bounding box prior has width ( $p_w$) and height ( $p_h$), then the predictions are as follows:

$$b_x=\sigma (t_x)+c_x$$

$$b_y=\sigma (t_y)+c_y$$

$$b_w=p_w\cdot e^{t_w}$$

$$b_h=p_h\cdot e^{t_h}$$

$$\mathrm{P}\mathrm{r}(\mathrm{o}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t})\cdot \mathrm{I}\mathrm{O}\mathrm{U}(b,\mathrm{o}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t})=\sigma (t_o)$$

Object detection, distinct from image classification, grapples with the challenge of identifying and precisely localizing multiple objects, potentially belonging to various classes, within a single image. The primary objective is to accurately predict the presence and location of all these objects. The subsequent steps outline the process involved in anchor-free detection.

• Center points prediction: In anchor-free methods, the network predicts the center points of potential objects. This is often represented as a heat map where each point in the heat map corresponds to a potential object center.

Mathematically:

Let us denote the heat map for object center points as C, and $C(x,y)$ represents the confidence score of an object center at location $(x,y)$.

• Bounding box regression: Once the center points are predicted, anchor-free methods perform bounding box regression relative to these centers. They predict the offset ( $\mathrm{\Delta }x,\mathrm{\Delta }y$) for each object center, which determines the position of the bounding box around that center.

Mathematically: For a center point $(x,y)$, the bounding box position can be represented as:

$(x+\mathrm{\Delta }x,y+\mathrm{\Delta }y)$

• Objectness score: To estimate if a bounding box indeed contains an object, an objectness score is predicted. This score is high for boxes containing objects and low for empty ones.

  Mathematically: The objectness score for a bounding box can be represented as $\sigma$, and $\sigma (x,y)$ represents the probability that the bounding box at $(x,y)$ contains an object.

• Final bounding box prediction: The final bounding box for an object is determined by adding the predicted offset to the center point. The size of the bounding box can also be a part of the prediction.

  Mathematically:

  The coordinates denoting the top-left and bottom-right corners of the bounding box can be expressed as follows:

  • –

    Top-left corner: $$(x+\mathrm{\Delta }x-\frac{w}{2},y+\mathrm{\Delta }y-\frac{h}{2})$$

  • –

    Bottom-right corner: $$(x+\mathrm{\Delta }x+\frac{w}{2},y+\mathrm{\Delta }y+\frac{h}{2})$$

Here, $(x,y)$ is the predicted center, $\mathrm{\Delta }x$ and $\mathrm{\Delta }y$ are the predicted offsets, and $(w,h)$ represents the width and height of the bounding box.

Configuration

To facilitate comprehensive analysis in our research, we leverage the MS COCO dataset (Lin et al., 2014). The Common Objects in Context (COCO) dataset, developed by Microsoft and extensively documented, encompasses training, validation, and test sets comprising over 200,000 images distributed across 80 distinct object categories. This dataset serves as a comprehensive resource for object detection and image segmentation tasks. Figure 5 provides a visual representation of the various object classes present in the COCO dataset, offering a rich and diverse collection of images for training and evaluating computer vision models. Developed to advance object recognition and scene understanding, the COCO dataset has become a benchmark in the field, providing a standardized foundation for researchers and practitioners working on image-related tasks.

In configuring the model for our experiment, as illustrated in Table 1, we meticulously defined a set of parameters crucial to the training and performance of the neural network. The training process spanned $200$ epochs, each representing a complete iteration through our dataset. To guide the optimization process, we employed a learning rate of $0.01$, striking a balance between convergence speed and precision. Input images were standardized to dimensions of $512\times 512$ pixels, and training occurred in batches of $16$ images at a time, facilitating efficient parameter updates. Our comprehensive dataset, comprised of $26,520$ images, served as the foundation for model learning.

The architecture, consisting of 225 layers, was meticulously chosen to capture intricate patterns in our data. With a total of $11,136,374$ parameters, encompassing weights and biases, our configuration reflects a thoughtful balance between model complexity and computational efficiency, ultimately tailored to the specific demands of our research context.

Performance metrics

The F-measure (FM) or F-1 score is determined by the weighted average of recall percentages and precision percentages. This score, also known as the harmonic mean of precision and recall, takes into account both false positives and false negatives. While FM is more commonly used than precision alone, accuracy is not immediately straightforward to comprehend. Accuracy performs well when false positives and false negatives have comparable costs. However, if the costs associated with false positives and false negatives differ, consideration of both recall and accuracy is preferable.

In the context of positive findings, precision (P) represents the proportion of accurately predicted observations to all predicted positive findings. On the other hand, recall (R) is the proportion of true positive predictions over all actual positives. These metrics can be calculated as follows:

(3) $$P=\frac{TP}{TP+FP}$$

(4) $$R=\frac{TP}{TP+FN}$$

In standard terminology, TP represents true positive, FP stands for false positive, TN denotes true negative, and FN refers to false negative. Precision and recall are taken into account when calculating the F-measure (FM), abbreviated as FM and computed using the following formula:

(5) $$FM=\frac{2\cdot R\cdot P}{R+P}$$

In addition to various other evaluation metrics, we incorporate the accuracy metric to evaluate the overall correctness of the model’s predictions. The accuracy metric offers insight into the proportion of correctly classified predictions among the total number of object predictions. Its calculation is determined by the following formula:

(6) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{TP+TN}{TP+TN+FP+FN}$$

Accuracy is a frequently employed performance metric, particularly in situations where the dataset is well-balanced, indicating a relatively equal number of positive and negative predictions. This metric provides a simple and direct measure to assess the correctness of the model in classification tasks. In addition to all other metrics, frame per seconds (FPS) is also used as the evaluation metric and model performance is judged accordingly.

Average precision (AP) is a widely utilized metric in object detection, evaluating a model’s accuracy in detecting objects across different levels of precision. AP quantifies performance by computing the area under the precision-recall curve (AUC − PR) at various thresholds. The formula for calculating AP is expressed as:

(7) $$AP=\sum _{i=1}^n(R_i-R_{i-1})\cdot P_i$$

Here, AP represents the average precision, $n$ denotes the number of data points, $R_i$ stands for recall at point $i$, $R_{i-1}$ represents recall at the previous point, and $P_i$ represents precision at point $i$.

Mean average precision ( $mAP$) serves as an indicator of a model’s comprehensive performance across all classes in object detection. It is determined by averaging the individual average precisions ( $APs$) computed at various precision levels. The calculation of $mAP$ can be expressed using the formula:

(8) $$mAP=\frac{1}{N}\sum _{i=1}^{N}A{P}_i$$

In this formula, $mAP$ represents the mean average precision, N denotes the number of classes, and $APi$ signifies the average precision for each class $i$.

Results and Discussion

Our IR-DD model training and testing were accelerated by the formidable NVIDIA RTX 2080Ti GPU and the Intel $I9-10900X$ CPU running at $3.7GHz$, resulting in substantial performance improvements.

Table 2 offers a comparative analysis of diverse object detection methods evaluated on the MS-COCO dataset (Youssouf, 2022; Ye et al., 2023; Mahendru & Dubey, 2021; Guo et al., 2022; Qu et al., 2022; Dong, Tang & Zhang, 2023; Sirisha & Sudha, 2022), presenting key performance metrics. The evaluation metrics, including precision, recall, F1 score, mean average precision (mAP) at IoU thresholds of 0.5 and 0.5:0.95, along with frames per second (FPS) during inference, offer a comprehensive assessment of each method’s effectiveness. Notably, our proposed method achieves compelling results with a precision of 0.666, F1 score of 0.630, mAP@0.5 of 0.650, and an impressive FPS of 86. These metrics collectively underscore the competitive performance of our method in object detection, demonstrating proficiency across precision, recall, F1 score, mAP, and computational efficiency on the MS-COCO dataset.

Table 3 and Fig. 6 provides a comparative performance overview for various iterations of the YOLO object detection algorithm, evaluating their effectiveness through two key metrics: frames per second (FPS) and average precision at an IoU threshold of 0.5 (AP50). YOLOv8 emerges as the fastest, boasting an impressive 280 FPS; however, this efficiency comes at a cost, as it registers the lowest AP50 at 53.9. In contrast, YOLOv6-L strikes a notable balance between speed and accuracy, achieving the highest AP50 of 70 while maintaining a respectable FPS of 98. YOLOv5-L also performs competitively, with an FPS of 113 and an AP50 of 67.3. These metrics offer practitioners valuable insights, emphasizing the nuanced trade-off between speed and precision when selecting the most suitable YOLO version for specific object detection applications. Notably, the utilization of the YOLOv8 model in our road distress detection system is strategically grounded in its superior performance metrics, striking a well-calibrated balance between rapid processing capabilities and precise object localization.

The Fig. 7 describes a collection of illustrative instances highlighting the results of object detection using our model on the MS-COCO dataset (Lin et al., 2014). These instances, presented in the form of figures, showcase the model’s performance in accurately detecting and delineating various objects within different pictures. The use of YOLOv8, known for its efficiency in real-time object detection, emphasizes the effectiveness of the model in identifying and localizing objects across diverse scenarios present in the MS-COCO dataset. These visual representations serve as concrete examples of the model’s capabilities and contribute to a more tangible understanding of its object detection performance.

In our study, the integration of the EfficientNet as a backbone, coupled with the utilization of BiFPN, has proven to be a strategically advantageous choice for enhancing the performance of our model. EfficientNet, known for its superior scaling properties and efficient use of model parameters, serves as a robust foundation for feature extraction, enabling the network to capture intricate patterns and representations within the input data. The incorporation of BiFPN further contributes to improved feature integration and information flow across different network layers, fostering enhanced contextual understanding. The bidirectional connections in BiFPN facilitate effective communication between feature maps at various scales, promoting richer feature representations crucial for accurate object detection. The synergistic combination of EfficientNet and BiFPN, as demonstrated in our experimental results, not only optimizes model efficiency but also bolsters the overall precision and recall of our object detection system, making it well-suited for diverse and complex visual scenarios. This strategic architectural choice underscores the significance of leveraging state-of-the-art components to achieve superior performance in computer vision tasks.