DES-YOLO: a novel model for real-time detection of casting surface defects

DSC-Darknet backbone network

The CSP-Darknet53 backbone network is designed to leverage its residual structure to move convolutional layers across the original image, facilitating the fusion of various fine-grained image features. Our investigations have indicated that applying CSP-Darknet53 to detecting defects on casting surfaces does not markedly elevate detection precision; instead, it results in heightened computational load and extended training durations. We propose the DSC-Darknet feature extraction network to improve the accuracy of casting surface defect inspection models.

In the DSC-Darknet backbone network, traditional convolution layers are replaced with DSC convolutional layers, comprising Distributed Shift Convolution (DSConv (Gennari, Fawcett & Prisacariu, 2019)), batch normalization (BN), and the CELU activation function. DSConv, functioning as a plug-and-play convolution, employs a variable quantization kernel (VQK) and dual distribution shifts to substitute standard convolution layers in CNNs seamlessly. The VQK is engineered to store integer values of variable bit-length, thus minimizing floating-point calculations, reducing error accumulation, and lowering computational overhead. The distribution shift mechanism is segmented into the kernel shifter (KDS) and the channel distribution shifter (CDS). KDS adjusts the depth value of the VQK for each segment, while CDS consolidates channel values per segment. This structure enables the kernel-based and channel-based distribution shifters to replicate the outputs of the original convolution, which accelerates network convergence and enhances the model’s generalization capabilities.

The DBC-Block design, illustrated in Fig. 2B, incorporates DConv and partitions the feature map into two divided into two independent branches. The primary pathway transmits features directly to the Conv layer, while the auxiliary path aggregates high-frequency features through the Conv and DSConv layers. Two independent branches implement feature aggregation at different scales using the Conv layer. This structure of the DBC-Block effectively addresses potential issues of gradient vanishing associated with increased network depth and simplifies the overall complexity of the DSC-Darknet backbone network.

DSC-Darknet generates feature information across five scales via a cascading feature transfer. As a high feature layer, P5 features rich semantic and contextual information but will lose the characteristic information of small defective targets. To address this, a global attention mechanism (GAM attention; Liu, 2021) is introduced following the P5 layer. This mechanism is designed to refine the detection of small targets by reducing background noise and evenly distributing feature data within the spatial domain. The GAM attention mechanism consists of two submodules: channel attention and spatial attention, illustrated in Fig. 3. The channel attention component employs a multilayer perceptron (MLP) to redistribute feature information and underscore interdimensional relationships. The spatial attention submodule employs two convolutional layers aligned with channel data to enhance spatial details and focus on more minor target features.

Enhanced pyramid pool module

Utilizing multiscale feature extracts within the DSC-Darknet framework may lead to reduced contextual information and the loss of fine-grained features across various subregions. The enhanced pyramid pooling module (EPPM) is integrated into the last layer feature map to address this challenge. The EPPM is composed of the multilevel pyramid pooling model (MPPM) (Zhao et al., 2016) and the enhancement module (EM) (Yu et al., 2022). The model polymeric prior global contextual information enhances the flexibility of the feature extraction network and improves its ability to detect small targets.

The MPPM module addresses the loss of spatial information attributable to average pooling by using (2  × 2, 3  × 3, 5  × 5, 6  × 6) four-dimensional pooling branch to segment the feature map into distinct subregions, as depicted in Fig. 4A. To preserve global contextual details, the MPPM conducts convolution operations that reduce the contextual dimensions to a quarter of their original size following multilevel pooling. Each branch feature value is processed through the ReLU activation function to enhance the model’s nonlinearity. The MPPM module utilizes bilinear up-sampling to return low-level features to their original dimensions to aggregate features without adding computational overhead. Ultimately, The MPPM module utilizes a cascading technique to integrate global contextual information with the multidimensional features of the local context, thereby enhancing the generation of robust global features, as detailed in Eq. (1). The input feature F_input generates four pooled features F_pooling with different dimensions after average pooling and 1  × 1 convolution operations with different bin values are set. The MPPM module integrates multiscale features by up-sampling and concatenation operations on branches, and achieves channel dimensionality reduction by 1  × 1 convolution operations to generate multiscale features F_MMPM. (1)${\displaystyle \begin{array}{c}{F}_{pooling\text{\_}i}=C\left(\left(P\left(F_{\text{input}}\right)\right)\right)\hfill \\ F_{MPPM}=C\left(U\left(F_{pooling\text{\_}1}\right)+U\left(F_{pooling\text{\_}2}\right)+U\left(F_{pooling\text{\_}3}\right)+U\left(F_{pooling\text{\_}4}\right)\right)\hfill \end{array}}$

U(X) represents a bilinear interpolation up-sampling operation applied to the feature map. C(X) denotes a convolution operation with a kernel size of 1 and a step size of 1, while P(X) stands for the average pooling of the feature map. The utilization of the average pooling preserves finer details in the image information, effectively addressing the challenge of increased variance in estimation values resulting from restricted neighborhood sizes.

The EM module incorporates convolution branches that feature varying expansion rates, residual links, and stratified weighting to expand the feature receptive field and address challenges such as reducing detail in small target features, gradient instability, and signal attenuation during the training phase. The multi-scale features of the MPPM part are passed through a convolutional layer with a kernel size of 1 to reduce the number of feature channels by half.

The multi-expansion rate convolution branch within the EM is designed to augment the semantic expressiveness of the receptive field without altering its size. This branch is structured with a three-branch structure, utilizing 3  × 3 convolutions and dilation convolutions with expansion coefficients of 1, 2, and 3. This arrangement conserves computing resources and diminishes the risk of overfitting through a weight-sharing strategy among the branches. Subsequently, the channel dimension of the feature map is enriched by a 1  × 1 convolutional layer, which ensures the restoration of the enhanced feature channels to the original image’s dimensions while concurrently preserving detailed feature information, as delineated in Eq. (2). Where the symbol “+” stands for branch aggregation. The enhancement feature F_EN fuses the multiscale feature F_MPPM and its cubic convolutional features by residual concatenation. And by applying 1  × 1 convolution and pooling operation on the multi-branch enhancement features, the channel dimension is unified, which in turn enhances the feature representation of defective sites.

The EM is instrumental in broadening the receptive field, reducing overfitting, and optimizing the representational capacity of samples across various scales. (2)${\displaystyle \begin{array}{c}{F}_{EN\text{\_}i}=F_{MPPM}+C\left(D_{\mathrm{i}}\left(C\left(F_{MPPM}\right)\right)\right)\hfill \\ F_{EPPM}=P\left(C\left(F_{PPM},F_{EN\text{\_}1},F_{EN\text{\_}2},F_{EN\text{\_}3}\right)\right).\hfill \end{array}}$

Slim-neck fusing GSConv and VoV-GSCSP

Extensive computation in casting surface defect detection results in prolonged training time, posing challenges for practical implementation. The DES-YOLO adoption of a slim convolutional architecture known as GSConv (Li et al., 2022) is proposed to mitigate this. GSConv combines multiple convolutional operations, including standard convolution (Conv), depth-wise separable convolution (DSConv), and a Shuffle mechanism, as illustrated in Fig. 5A. This architecture reorganizes the input feature channels using Conv to facilitate the computation of features in multiple groups. It replaces the standard Conv of each group with DSConv and enhances inter-group feature integration by shuffling channel features. With a convolution kernel dimension of K1 ×K2, input feature map channels of Ci, output feature map channels of Co, and an output image dimension of Wo × Ho. Traditional convolution has a single computation of K₁ × K₂ × C_i × C_o × W_o × H_o, while DSConv convolution is divided into two parts: channel-by-channel convolution and point-by-point convolution. Channel-by-channel convolution uses only one convolution kernel per channel and has K₁ × K₂ × C_i × W_o × H_o parameters, while point-by-point convolution uses 1  × 1 standard convolution of features and has C_i × C_o × W_o × H_o parameters.The computational load of GSConv is significantly reduced in comparison to traditional Conv, as demonstrated in Eq. (3). (3)${\displaystyle \frac{P_{DSConv}}{P_{Conv}}=\frac{K_1\times K_2\times C_i\times W_o\times H_o+C_i\times C_o\times W_o\times H_o}{K_1\times K_2\times C_i\times C_o\times W_o\times H_o}=\frac{1}{C_o}+\frac{1}{K_1\times K_2}}$

The VoV-GSCSP module, inspired by the ELAN architecture in YOLOv7, employs an unilayered aggregation strategy, as depicted in Figs. 5B and 5C. The VoV-GSCSP module uses dual GSConv to replace the traditional convolution operation. It uses the residual structure to sequentially splice shallow features with strong multi-scale positioning capabilities and deep features with strong multi-scale semantic expression capabilities into features that fuse multi-layer information. Vectors avoid the problems of information loss and gradient disappearance and reduce the complexity of calculation and network structure.

Casting surface defect dataset -CSD DET

According to the Chinese national standard GB/T5611, casting surface defects are primarily classified into three categories: hole defects, incomplete defects, and surface defects, with each category further divided into seven subcategories. Hole defects arise from bubbles formed during casting when gas fails to escape during solidification. These are subdivided into blowholes, shrinkage cavities, and slag holes, distinguished by their visual characteristics. Insufficient metal liquid pouring or mechanical collision falls under incomplete defects. Surface defects, referring to general appearance issues in castings, are typically categorized into hollows and scratches, each resulting from varied manufacturing anomalies.

This study collected images of surface hole defects in textile machine parts, furnace castings from a hardware foundry in Xuancheng City, Anhui Province, and automobile cylinder block castings produced by an automobile industry company in Wuhu City.

A Hikvision industrial camera was used to capture these images, allowing for the simulation of diverse production environments through adjustments to the camera aperture and background images. The camera features a maximum frame rate of 44.7 fps, an eight mm focal length lens, and an aperture range of F2.4 to F16. Images were captured at a resolution of 1,280 pixels × 1,280 pixels, with a shooting interval of 0.2 s. During data acquisition, special consideration was given to the defect samples’ scale, angle, and viewpoint diversity to adapt to variations in target defects under different distances and sizes, aiming to improve the model’s accuracy in defect detection and recognition. After a rigorous filtering process, 5,390 images of casting surface defects were obtained, each containing at least one defect—analysis of the collected images with 15% featuring multiple defects of varying sizes and uneven distribution. Multiple rounds of manual review were conducted to address the challenge of accurately labeling defect locations and categories in the dataset to ensure labeling accuracy and consistency. Additionally, the reliability of the labeling results was verified through random sampling and repeated labeling of selected data.

An analysis of category distribution within the dataset was conducted to address the imbalance in data samples caused by the low occurrence rates of shrinkage and hollow defects, and the sample numbers were balanced using random coefficients data enhancement techniques. These augmentation methods, including random luminance contrast adjustment, grid distortion, and affine transformations, were utilized to reduce data imbalance, as illustrated in Fig. 6. After applying these data enhancement techniques, the casting surface defect dataset, now encompassing 6,000 images with 8,854 defect markers, was designated as CSD-DET, as detailed in Table 1. Domain experts in casting were then engaged to evaluate the practical utility of the CSD-DET dataset for detecting surface defects in castings.

Loss functions

During the training of the DES-YOLO model, defect prediction frames are continuously generated. This study leverages the comparison between these prediction frames and ground truth (GT) frames to evaluate the model’s ability to detect and accurately predict real defects. To achieve this, the SIOU loss function is utilized to optimize model parameters and enhance prediction accuracy by minimizing the loss. The SIOU loss function is employed, incorporating components such as confidence loss, coordinate regression loss, and target confidence loss, as detailed in Eq. (4). Here, Loss_{classification} denotes the category confidence loss, which is used to determine whether the anchor frame is correctly classified. Loss_localization denotes the coordinate regression loss, which is used to measure the error between the prediction frame and the real frame. Loss_confidence denotes the target confidence loss, which evaluates whether the prediction frame contains the target or not. λ is used as a super parameter to balance the effect of different loss functions on the backpropagation of model parameters by weighting coefficients,Throughout the model optimization process, with manual adjustments made multiple times and set to λ1 = 0.1, λ2 = 0.005, and λ3 = 0.1 to meet the performance optimization requirements. The calculation of target and class confidence losses employs Binary Cross Entropy with Logits Loss (BCE With Logits Loss) combined with the sigmoid function. (4)${\displaystyle Los{s}_{all}={\lambda }_{1}Los{s}_{classification}+{\lambda }_{2}Los{s}_{localization}+{\lambda }_{3}Los{s}_{confidence}}$

The YOLO model typically employs the Complete Intersection over Union (CIoU) as the coordinate regression loss function. This function facilitates prediction and refinement by evaluating distances and overlap areas between the predicted bounding boxes and the ground truth (GT) boxes. While the CIoU loss function aids in model optimization, it lacks consideration of the directional relationship between predicted boxes and GT boxes, resulting in the slow convergence speed of the model.

The DES-YOLO study employs the SIoU (Zhora, 2022) loss function, incorporating an area proportionality penalty term and a vector angle between projective regression. It comprises angle cost, distance cost, shape cost, and IoU cost, as depicted in Eq. (5). Where Δ represents the angular and distance loss coefficient and Ω represents the shape loss coefficient, IoU represents the shape loss coefficient. (5)${\displaystyle Los{s}_{SIoU}=1-IoU+\frac{\Delta +\Omega }{2}}$

To ensure smooth and controllable model convergence, an exponential decay function is employed to calculate the shape distance loss between predicted and actual frames, incorporating angular loss coefficients. The distance loss function Δ is associated with the minimum outer rectangle of the two bounding boxes, as defined in Eq. (6). Where Λ denotes the angular loss coefficient, this coefficient is calculated by determining the minimum angles α and β between the connecting line from the center point of the predicted bounding box to the actual bounding box and the x and y axes. The angular loss coefficient guides the predicted bounding box towards the direction of the axis of the nearest target box when the center point is not aligned with any of these axes (Λ ≠2). The Δ loss is constrained to 0 when the center point of the predicted frame aligns with either the x-axis or y-axis, specifically when (Λ =2). ($b_{cx}^{gt},b_{cy}^{gt}$) represents the center coordinates of the ground truth (GT) box, and (b_cx, b_cy) represents the center coordinates of the predicted box. Additionally, c_w and c_h indicate the width and height of the minimum bounding rectangles for both the GT and predicted boxes, as illustrated in Fig. 7B. Equation Λ Through the improved distance calculation formula, the angle α between the real frame and the center point of the prediction frame is determined, and the angle loss value is estimated using the multiplier angle formula. (6)${\displaystyle \begin{array}{c}\Lambda =1-2\times sin2\left(arcsin\left(\frac{max\left(b_{cy}^{gt},b_{cy}\right)-min\left(b_{cx}^{gt},b_{cx}\right)}{\sqrt{{\left(b_{cy}^{gt}-b_{cy}\right)}^2+{\left(b_{cx}^{gt}-b_{cx}\right)}^2}}\right)-\frac{\Pi }{4}\right)\hfill \\ \Delta =\sum _{\mathrm{t}=x,y}\left(1-e^{\left(\Lambda -2\right){\rho }_t}\right)=2-e^{\left(\Lambda -2\right){\left(\frac{b_{cx}^{gt}-b_{cx}}{c_w}\right)}^2}-e^{\left(\Lambda -2\right){\left(\frac{b_{cy}^{gt}-b_{cy}}{c_h}\right)}^2}\hfill \end{array}}$

In the shape loss function, the value of θ determines the unique Shape cost for each data point, as indicated in Eq. (7). The value of θ affects the gradient update step for the model’s prediction box shape loss, and setting θ to 3 accelerates model training during the experimental manual parameter tuning process. The shape loss coefficients accumulate prediction errors across coordinates using an exponential decay model. (7)${\displaystyle \Omega =\sum _{\mathrm{t}=w,h}{\left(1-e^{-w_t}\right)}^{\theta }.}$

The width and height of the predicted and ground truth boxes are denoted by x and y, respectively, with the distance between their center points represented as d and the convergence speed as c. As depicted in Fig. 7A, the prediction frame shifts in the X-axis direction, progressively nearing the GT frame. The overlap of CIoU after the first convergence is denoted as dc, and after the second convergence, it becomes 2dc. In contrast to CIoU’s linear movement along the x-axis, SIoU steers the prediction frame toward the line connecting the center points of the predicted and ground truth frames. During this process, SIoU dynamically modifies the bounding box’s dimensions to align with those of the ground truth box. Figure 7B illustrates the overlap degree between the two boxes after the two convergences.

The overlap degree for SIoU after the initial convergence is calculated as $\frac{xy{c}^2}{x^2+y^2}$ , and it increases to $\frac{4xy{c}^2}{x^2+y^2}$ after the second convergence. Compared to CIoU, the overlap degree after movements guided by SIoU is significantly higher. This efficiency in diagonally directing the predicted bounding box’s center point toward the real bounding box’s center point facilitates faster model convergence during optimization.

Experiment environment and evaluation index

Ablation experiments

This section details ablation experiments conducted to evaluate the effectiveness of individual modules within the DES-YOLO model for casting surface defect detection. The results are presented in Table 2, where the “✓”symbol denotes the inclusion of specific modules in the experiment. F1-score as an aggregate function of precision and recall. In the process of gradually introducing each module in the ablation experiment, the F1-score increased from 88.1 to 89.0, which proves that each module in the DES-YOLO model minimizes background misrecognition and enhances the accuracy of the correct identification of defective targets. The implementation of the DSC-Darknet referred to as A1, shows a detection accuracy that surpasses YOLOv7-tiny by 2.3% and 1.5%, underscoring the enhanced feature extraction capabilities of the DSC-Darknet backbone network.

Additionally, the inclusion of the GAM Attention and EPPM modules significantly boosts detection accuracy. While the slim-neck module enhances accuracy at the IoU threshold 0.5, a slight reduction is noted at the 0.5:0.95 IoU threshold. This decrease is attributed to the slim-neck’s shallow network depth, which, while reducing the computational load, aligns with practical application demands. Integrating the SIoU metric into the DES-YOLO model results in a mAP at 0.5 of up to 92.6% and a mAP at 0.5:0.95 of up to 68.8%, confirming the method’s efficacy in identifying defects on casting surfaces.

While an attention mechanism is not imperative for target detection, experiments reveal that its application significantly enhances the model’s ability to detect small target features, achieving accurate surface defect detection in inspection castings. Comparative analyses with mainstream mechanisms, such as ECA and NAM Attention, demonstrate the superior detection performance of GAM Attention, even without a substantial increase in parameter numbers. Although SK Attention exhibits the highest detection accuracy, its extensive requirement for calculating location information renders it less suitable for lightweight applications within factory settings. Notably, in the GAM attention experiment, accuracy rates exceed recall rates, a phenomenon that may be influenced by noise in the casting defect samples. This observation is corroborated by Fig. 8, which confirms that GAM attention is the most suitable for the DES-YOLO model.

The primary objective of the casting surface defect detection task is to effectively differentiate between background elements and characteristic features within the castings. A saliency heatmap is employed to demonstrate the detection model’s focus on defects visually. This heatmap uses warmer image tones to represent higher model attention, indicating a greater probability that the highlighted region harbors defective features. Figure 9 illustrates how the model’s attention to defective surface areas becomes more pronounced throughout the iterative training process.

In order to verify that the DES-YOLO model can effectively detect casting surface defects under different environments and lighting conditions, this study deployed the model in casting production line conveyor belts, casting storage warehouses, and other production environments. The test results are shown in Fig. 10, which proves that the DES-YOLO model has high generalization ability in different environments.

Comparative experiments

To evaluate the improvement of the DES-YOLO model, this study presents a comprehensive comparison with the mainstream model, as summarized in Table 3. Compared to Faster R-CNN and RetinaNet, the YOLO series algorithms demonstrate significantly fewer parameters. Despite its higher parameter count, Faster R-CNN, a two-stage algorithm, attains the highest detection accuracy for scratch category defects at 85.1%. This superior performance is attributed to its Pyramid Region Network (PRN) characteristics and two-stage processing, effectively detecting minor target defects. In contrast, YOLOv3 demonstrates the lowest detection accuracy, with mAP’s IoU threshold at 0.5 of only 84.3% and mAP’s IoU threshold at 0.5:0.95 of 56.6%. The DES-YOLO model achieves a mAP’s IoU threshold of 92.6%, reflecting a 5.3% improvement over the YOLOv7 model’s 87.3% accuracy, primarily due to the utilization of DSC-Darknet and the EPPM model. The DES-YOLO model excels in detecting four types of defects: blowholes, slag holes, fractures, and misruns, with visualization results presented in Fig. 11.

To assess the DES-YOLO model’s suitability for industrial real-time detection, its inference speed and computational complexity were analyzed and compared with other models, as detailed in Table 4. Faster R-CNN and RetinaNet lag behind the YOLO series regarding inference speed and computational complexity, posing challenges for industrial real-time detection. Although YOLOv3-tiny offers lower computational complexity, it does not achieve the inference speed or detection accuracy of the models discussed in this study.

In comparison, YOLOv5-s, YOLOR-p6, PPYOLOE-s, and YOLOX-s all demonstrate slower inference speeds, with their GFLOPs exceeding those of our model by 20.4%, 1.5%, 27.0%, and 95.6%, respectively. Notably, the DES-YOLO model achieves an inference speed of 3.9 ms, 0.5 ms faster than YOLOv7-tiny, while maintaining a computational load of only 13.7 GFLOPs. This efficiency satisfies the stringent requirements for real-time detection of casting surface defects. Analysis of the loss attenuation curves, as shown in Fig. 12, reveals that the DES-YOLO model’s confidence loss relative to total loss declines more smoothly and rapidly than YOLOv7-tiny, indicating enhanced learning and generalization capabilities.

The analyses above further corroborate that our proposed DES-YOLO model exhibits superior robustness in the real-time detection of casting surface defects.