Adaptive visual detection of industrial product defects

Traditional visual detection methods

Traditional visual detection methods mainly relies on manually designed features to complete defect recognition, which can be divided into two categories: the texture feature-based methods and the color feature-based methods.

• The texture feature-based methods: The texture feature-based methods are mainly based on the grayscale distribution of pixels and their spatial neighborhoods for anomaly detection, which can be further classified into three categories: statistical methods, signal processing methods, and model methods. For statistical methods (Song et al., 2015), the main idea is to describe various statistical characteristics of the distribution of gray values. Then detection is performed by artificially setting thresholds. The signal processing methods (Tsai, Wu & Li, 2012) regard the image as a two-dimensional signal and analyze the image from the perspective of signal filter design. The traditional modeling method is to build detection models for specific tasks, and the common detection methods are mainly MRF (Markov random field) models (Luthon, Popescu & Caplier, 1994) and fractal models (Xu, 2015) etc.

• The color feature-based methods: Stain defects on the surface of industrial products can cause color jumps. The color feature-based methods mainly uses color histograms (Prasitmeeboon & Yau, 2019) to describe the proportion of different colors in an image or color moments (Li, Quan & Wang, 2020) to describe the distribution of colors to accomplish anomaly detection.

Traditional defect detection methods are mainly designed for specific tasks, and the model has poor generalization and flexibility. The feature extraction strategy based on manual setting is unstable and cannot effectively deal with the complex industrial production environment.

Deep learning detection methods

Deep learning applies the end-to-end training strategy to enable the model to learn the defect features automatically, which can enhance the model’s generalization performance. Deep learning based methods for industrial defect detection can be divided into two main categories: supervised learning and semi-supervised or unsupervised learning.

• Supervised learning: Supervised industrial defect detection methods are mainly applied to situations where the defect patterns are known. The most direct and simple way is to apply CNN-based classification (Zhang, Gu & Zhang, 2021), detection (Li et al., 2021), and segmentation (Long, Shelhamer & Darrell, 2015) models to industrial defect detection tasks. The classification of the appearance quality of industrial products is the simplest and most straightforward visual inspection task. However, for some special applications, it is necessary to obtain the position information of industrial product defects, even the contour information, to improve the production process. Representative works can refer to Villa et al. (2018) and Jing et al. (2020).

• Semi-supervised or unsupervised: In practical industrial production, it is not easy to obtain enough defect datasets. Even the occurrence of zero-sample unknown defects is possible. The semi-supervised and unsupervised industrial defect detection methods mainly applies to the situations with unknown defects. The unsupervised industrial defect detection methods mainly rely on generative models, which believe that generators built with known samples cannot produce satisfactory results for the discriminator when encountering unknown defects. The most commonly applied reconstruction methods are autoencoder (AE) (Alahmadi, Alkhraan & BinSaeedan, 2022) and generative adversarial network (GAN) (Arora & Soni, 2021). The semi-supervised industrial defect detection methods, from a statistical point of view, consider that the distribution of abnormal samples in the feature space is inconsistent with that of normal samples. In high-dimensional space, or abstract feature space, unknown sample detection can be achieved by setting the data distribution discriminator. Typical methods can be referred in SPADE (Cohen & Hoshen, 2020), PADIM (Defard et al., 2021).

Industrial defect detection models pursue application generalization and rapidity. The strong adaptive feature extraction network for industrial defects is the key. To the best of our knowledge, there is currently no research focusing on general-purpose feature extraction models for industrial defects.

CNN and meta learning

With the increase in the amount of data and the improvement of computing power, deep neural networks (DNNs) have shown strong performance in the field of machine learning. CNN is the most typical representative of DNNs, which can allow image-level input and realize local feature extraction of images. Feature extraction based on convolution operation is the most important component in CNN. In a specific layer of convolution operation, the weight-shared convolution kernel traverses the entire image, and serves as a feature extractor to highlight local target features. The training process of CNN model is the process of determining parameters of convolution function, which is composed of the trainable parameters $w=\left(w_1,w_2,\cdot \cdot \cdot ,w_L\right)$ with random initialization. The trainable or learning parameters can be discriminatively determined by sample data $\left(x_i,y_i\right)$, for $i=1,2,\cdot \cdot \cdot ,n$, so that ${\sum }_n\ell \left(f\left(x_i,w\right),y_i\right)\to min$, where $\ell (\cdot )$ is the loss function. Convolutional layer is an important part of CNN. The convolutional layer usually consists of an input feature map, an output feature map, and a learnable convolutional kernel. The input feature map is a three-dimensional structure with a shape of $M\times N\times D$, and consists of D feature maps of size $M\times N$. The shape of the output feature map is similar to that of the input feature map, except that the size of each dimension may vary. Generally, one can using the back-propagation way to calculate the optimization gradients of learning parameters, and using a specific optimizer (e.g., Stochastic Gradient Descent or Adam) can lead CNN network gradually converge and obtain the satisfied learning parameters.

Meta learning is a type of transfer learning that aims to learn a general feature extractor from multiple datasets. Finn, Abbeel & Levine (2017) firstly proposed the ordinary MAML algorithm in 2017, which has undergone a great deal of follow-up research. Antoniou, Edwards & Storkey (2018) proposed MAML++, which improved MAML using an annealing algorithm to improve the generalization performance, convergence speed and computational power of MAML. Baik et al. (2021) made improvements to the loss function of MAML and proposed the first learnable loss function MeTAL, which enhanced the generalization ability of MAML. Zhou et al. (2021) theoretically derived an upper bound on the error of MAML and pointed out that the improvement in generalization performance for the target task is greater when the training task and the target task are more similar. Raghu et al. (2019) experimentally found that the generalization ability of MAML mainly comes from feature reuse, and proposed ANIL to substantially reduce the computational effort while ensuring generalization.

The models with CNN structure generally belong to the task-driven training methods, which seek to achieve satisfactory fitting accuracy under specific tasks. In contrast, the input of MAML is multiple task sets, and the training of MAML based models requires the balance of performance on multiple target tasks. MAML is the main training strategy of the proposed MeDetection model in this article, and the trained feature extraction network has a strong ability to extract general features for industrial defects.

Feature difference extraction module

It can be seen from Fig. 1 that the appearance defects of industrial products can be regarded as feature differences on the normal appearance, which gives us a conclusion to judge whether there are appearance defects on industrial products by detecting feature differences (Zhang et al., 2023). In the MeDetection model proposed in this article, the Siamese network serves the purpose of extracting difference features. Different from the traditional CNN structure, the Siamese network consists of parallel two-stream branches. As shown in Fig. 2, two branches input normal samples and test samples respectively, where the test samples contain normal and defective data. The feature extraction weights of the two branches are shared. The output of the Siamese network is obtained by the difference features of the two branches, such that

(1) $$x_o=abs\left(Con{v}_{\theta }\left(x_1\right)-Con{v}_{\theta }\left(x_2\right)\right)$$where $x_1,x_2$ represent the dual stream inputs, $x_o$ is the feature difference output. $Con{v}_{\theta }(\cdot )$ represents the feature extractor with $\theta$ as learning parameters, and $abs(\cdot )$ obtains the absolute values to ensure the non-negativity of the features.

Model training and optimization

Unlike task-driven pattern recognition models, the proposed MeDetection model is trained on multiple industrial defect visual inspection tasks. The types of industrial defects can be roughly divided into stains and texture defects, and the feature difference extraction module further improves the consistency of defect expression. We hope that the MeDetection model has generalized feature extraction capabilities and can adapt to different types of industrial defect recognition tasks.

The MAML-based training strategy is applied in MeDetection model. MAML seeks to achieve a balance of model performance on multiple task sets, that is, MAML does not seek to achieve the optimal model on a specific task, but hopes that the model can achieve comprehensive optimality on multiple task sets. The optimization function of MAML is as follows:

(2) $$L\left(f_{\theta }\right)=\sum _{i=1}^N{\ell }_{T_i}(f_{\theta }{}_i^{\mathrm{\prime }})$$where $L\left(f_{\theta }\right)$ represents the loss function of the model $f_{\theta }$ with $\theta$ as the learning parameter, which is equal to the sum of the loss functions $({\ell }_{T_i}(f_{\theta }{}_i^{\mathrm{\prime }}))$ of the model applied to all task sets $T_i,i=1,2,\cdot \cdot \cdot ,N$. ${\theta }_i^{\mathrm{\prime }}$ represents the task-specific weights generated by transferring the learning parameter $\theta$ to the $i$th task after fine-tuning operation. N is the number of tasks.

MAML consists of two loops: Inner loop and Outer loop. The optimal parameters for each task are calculated iteratively in the inner loop, while the outer loop updates the learning parameters for the entire model by computing the gradients relative to the optimal parameters in each new task. The pseudocode is shown in Algorithm 1. We select a variety of industrial products datasets to form the task sets $p(T)$. Specifically, $p(T)$ represents the probability distributions of all the training tasks (Finn, Abbeel & Levine, 2017). The learning rates of the inner loop and outer loop of MAML are set to $\alpha$ and $\beta$. During model optimization, the inner loop is executed firstly, and followed by the outer loop.

• Inner loop: We select K samples $D_i=\left\{x^{(j)},y^{(j)}\right\}$ from the task $T_i$. For classification task, ${\ell }_{T_i}$ is computed as follows:

(3) $$\begin{gathered} {\ell }_{T_i}(f_{\theta })=\sum _{x^{(j)},y^{(j)}\sim T_i}{y}^{(j)}\log f_{\theta }(x^{(j)}) \\ +(1-y^{(j)})\log (1-f_{\theta }(x^{(j)})) \end{gathered}$$

Then the updated gradients of model for the task $T_i$ can be computed as ${\mathrm{\nabla }}_{\theta }{\ell }_{T_i}\left(f_{\theta }\right)$. The learning parameters ${\theta }_i^{\mathrm{\prime }}$ for the task $T_i$ based on model $f_{\theta }$ can be obtained as:

(4) $${\theta }_i^{\mathrm{\prime }}=\theta -\alpha {\mathrm{\nabla }}_{\theta }{\ell }_{T_i}(f_{\theta })$$where ${\ell }_{T_i}(f_{\theta }{}_i^{\mathrm{\prime }})$ is calculated using the remaining samples $D_i^{\mathrm{\prime }}$ of the task $T_i$ based on Eq. (3).

• Outer loop: During the outer loop, the model parameters $\theta$ are updated using the following equation:

(5) $$\theta \leftarrow \theta -\beta {\mathrm{\nabla }}_{\theta }\sum _{T_i\sim p\left(T\right)}{\ell }_{T_i}(f_{\theta }{}_i^{\mathrm{\prime }})$$

Coordinate attention

The defect features extracted from the SN module may contain redundant information in the fusion process, which is not conducive to the recognition of defects by the model. CA (Hou, Zhou & Feng, 2021) constructs feature mapping from the horizontal and vertical dimensions, and interacts the feature information among channels. Through automatically re-weighting the features, CA module can highlight the defect features and eliminate the redundant information. As shown in Fig. 3, for the input $X=[x_1,x_2,x_3,...,x_c]\in {\mathbb{R}}^{C\times H\times W}$, CA uses two pooling kernels with dimensions of $(H,1)$ and $(1,W)$ to encode each channel along the horizontal coordinate and the vertical coordinate respectively. The output of the $c$-th channel at height $h$ and width $w$ can be expressed as

(6) $$\{\begin{array}{c}{z}_c^h\left(h\right)=\frac{1}{W}\sum _{0\le i<W}{x}_c\left(h,i\right)\\ z_c^w\left(w\right)=\frac{1}{H}\sum _{0\le j<H}{x}_c\left(j,w\right)\end{array}$$

The above two transformations enables the model to capture long-distance relationships in one direction while retaining spatial information in the other direction, which can help the network to identify defects more accurately. Then, CA module maps $z^h$ and $z^w$ to the corresponding coordinate attention weights, and reweighting operations are carried out.

The attention responses of CA module contain inter-channel information, horizontal spatial information and vertical spatial information, which can help the network to obtain the location information of defects more accurately and enhance the ability of feature extraction.

Dataset

The MeDetection model is able to learn common representations of defect features from multiple industrial defect tasks, and then implement knowledge transfer on new tasks. Therefore, we need to first prepare a cluster of industrial defect visual inspection task sets. In the experiments, DAGM (Wang et al., 2018) and MVTec (Bergmann et al., 2020a; Wieler & Tobias Hahn, 2007) datasets are applied in the training of MeDetection model. The MVTec dataset contains a total of 15 categories, five categories of which are texture-based data, and contain regular patterns (blankets, grids) and random patterns (leather, tiles, wood). The remaining 10 categories are object-based data, which contain objects with a specific appearance (bottles, metal nuts), deformable objects (cables) or objects including natural variation (hazelnut). Some of the acquired objects are in approximately aligned position (toothbrushes, capsules, and pill), the others were randomly placed (metal nuts, screws, and hazelnuts). The DAGM dataset contains 10 different types of fabric texture defects. Each class consists of 1,000 defect-free images and 150 defective images saved in grayscale eight-bit PNG format. The detailed information about the applied detect datasets is shown in Table 1.

In addition, in the previous work (Zhang et al., 2023), we published a novel visual dataset on appearance defects of industrial injection molding products, BC defects. There are 3,008 samples in BC defects dataset, including 1,608 normal samples and 1,400 ones with eight kinds of appearance defects.

Simulation details

The experiments run on a computer with NVIDIA DGX A100 SXM4 40G GPU. The experimental framework is pytorch1.9.0 with cuda 11.2. The image size is set to $256\times 256$. Adam with parameters ${\beta }_1=0.5$ and ${\beta }_2=0.999$ is used as the optimizer during training. We set the batchsize as 64. The max epoch is set as 1,000, and we use the early-stopping strategy to decide to stop the model training. The learning rates of inner and outer loops are set as $\alpha ={10}^{-5}$ and $\beta ={10}^{-3}$ respectively. Since the number of each industrial product is not unique, we first divide each task dataset into a support set and a query set according to 7:3, and then randomly select 64 images from the support set in the inner loop of MAML to train the model and then randomly select 20 images from the query set to validate the model. After MAML training, MeDetection will be fine-tunned on the target task with the initial learning rate as 0.0001. The advantage of the MeDetection model proposed in this article is that it can be trained on multiple industrial defect data sets and then transfer knowledge to unknown industrial defect recognition task. During training, we remove one target task from the data sets and train the model with the remaining industrial defect data sets. During the verification process, the trained MeDetection model is fine-tuned for the target task based on the pretrained learning parameters. In order to evaluate the model performance, we follow the metrics in Akcay, Atapour-Abarghouei & Breckon (2018) and compute the area under receiver operator characteristics (AUROC), which measures the area under the true positive rate as a function of the false positive rate. The AUROC metric is not sensitive to any threshold or the percentage of anomalies in the test set. In addition, other metrics, like Recall, Precision, F1 Score and the convergence speed of model, are taken into consideration for evaluating the model in all aspects.

Comparison experiments

In order to show the performance of MeDetection model more clearly, some excellent industrial defect detection algorithms are added to the comparative experiment. GANomaly (Akcay, Atapour-Abarghouei & Breckon, 2018) belongs to the semi-supervised defect detection via adversarial learning, which point out the unknown defect data lead to a larger distance metric from the learned (known) data distribution at inference time. One-NN (Nazare, de Mello & Ponti, 2018) represents the works that apply transfer learning technology into industrial defect detection, and it analyzes the usage of different feature normalization techniques on the pre-trained CNN models. DOCC (Ruff et al., 2021) is short for deep one-class classification method, which aims to deal with the unbalanced data distribution in industrial defect detection tasks. PatchSVDD (Yi & Yoon, 2020) is a long-standing algorithm used for an anomaly detection, which can search for a data-enclosing hypersphere in the kernel space, and compare the difference in data distribution between normal and defective samples. U-Student (Bergmann et al., 2020b) belongs to a powerful student–teacher framework for the challenging problem of unsupervised anomaly detection and pixel-precise anomaly segmentation in high-resolution images, where Student networks are trained to regress the output of a descriptive teacher network that was pretrained on a large dataset of patches from natural images. Anomalies are detected when the outputs of the student networks differ from that of the teacher network.

Table 2 shows the quantitative comparison results of AUROC metrics between the proposed MeDetection model with other state-of-the-arts on industrial defect detection tasks in MVTec (Bergmann et al., 2020b) datasets. The “Tasks” column represents the target task set, which needs to be eliminated during model training. MeDetection model gets the highest AUROC on six tasks (“carpet”, “leather”, “wood”, “capsule”, “pill” and “zipper”), the second best AUROC on three tasks (“grid”, “bottle” and “hazelnut”). Especially for the recognition task in “leather” dataset, MeDetection model gets 100% accuracy. The last column of Table 2 shows the average AUROC value. The MeDetection model achieves the best accuracy (94.0%), 1.5 percentage points higher than the U-Students model (Bergmann et al., 2020b), which achieves the second-best results in the overall comparison. In addition, we also present the weighted average results at the last row of Table 2, where the weights are determined according to the number of samples in the datasets. The larger the number of samples, the greater the weight. Due to the differences in the number of samples in the datasets involved in the simulation experiment, the results obtained by weighted average are more convincing. The proposed MeDetection model obtains the best performance with 91.8% weighted average recognition accuracy.

In addition, challenges are attached to the MeDetection model, and only 20 images in each training task participate in model training. The simulation results are shown in the last column of Table 2. The MeDetection model achieves suboptimal AUROC on three tasks (“carpet”, “carpet” and “pill”), and the average AUROC is higher than GANomaly (Akcay, Atapour-Abarghouei & Breckon, 2018) and 1-NN (Nazare, de Mello & Ponti, 2018) methods. It is worth mentioning that, the MeDetction model exhibits relatively weak recognition performance in the task of “screw”. The MeDetection model uses differential features as input, which has high requirements for target location alignment in dual channels. However, the orientation of the screws in the “screw” dataset are placed randomly as shown in Fig. 3, making it easy to generate pseudo-features when calculating differential features. Such deficiency is one of the future research directions of the MeDetection model.

In general, MeDetection has two advantages over other models. MeDetection has the highest accuracy and good detection capability for various industrial defects. MeDetection can use a small number of samples to train the model, which reduces the training cost and has a good detection accuracy at the same time.

Effectiveness of MAML

In order to verify the effectiveness of the MAML training training strategy in the proposed MeDetection model, we make comparison of the recognition accuracy based on random initialization of weights and MAML based weights transfer strategy. Table 3 shows the simulation results. The “Tasks” column represents the dataset used for model testing, while the remaining datasets are involved in the pre-training of MAML. The term of “random” in the column of “Weights” means that the weights used in the training of the recognition model are randomly initialized. Two absolute advantages for the MAML based training strategy for industrial defect detection can be obtained. Based on the MAML pre-trained weights, the training model has the relatively faster convergence rate. For the “fabric3” dataset, with the help of MAML pre-trained weights, the model only needs 50 epochs to converge, which is a quarter of the number of epochs of the model convergence under random weights. It is worth noting that with randomly initialized weights, the model fails to converge for the dataset of “capsule”. In addition, the MAML training strategy enables the model to improve the recognition performance in all five indicators (ACC, Recall, Precision, F1, AUROC). For the “BC defects” dataset, based on the MAML training strategy, the convergence efficiency of the model is more than doubled (450 epochs $\to$ 200 epochs). Meanwhile, the recognition performance has also been greatly improved. For the metrics of Precision, The MAML training strategy improves the recognition result by 18.8 percentage points. For the ACC, Recall, F1 and AUROC metrics, the recognition performance improved by about five percentage points. In addition, similar to Table 2, the weighted averages of AUROC scores are calculated in order to show the MeDetection model performance comprehensively. The weighted average of AUROC score of the model with randomly initialized weights is 94.7%, while the MeDetection model gets 97.6% weighted average AUROC score. We put the average results of each indicator in the last column of Table 3, which also demonstrates the performance of MeDetection model.

For industrial defect recognition tasks, we want to reduce the missed detection rate, that is, to avoid the flow of problematic products to the market. The recall rate can clearly evaluate the missed detection of the model. Figure 4 shows the curve of recall as a function of model training epochs on nine detection tasks. Models based on MAML pretrained weights not only converge faster, but also have higher recall than those trained with random weights.

Overall, the proposed MeDetection model exhibits very advanced recognition performance for industrial defect detection. The MAML based training strategy is the corner stone of MeDetection model, which brings two advantages: MAML can accelerate the convergence speed of the model well; MAML can improve the performance of the model greatly.

Defect location

The Grad-CAM algorithm (Selvaraju et al., 2017) can be used in the classification model to achieve target localization. For industrial defect detection tasks, obtaining the location information of defects is very helpful to analyze the causes of defects. Figure 5 shows the localization results of target defects, and the results of the model trained based on randomly initialized weights partipicate the comparison. The model with randomly initialized weights shows some deviations in the location of the defects, especially in the datasets of “capsule”, “fabric1” and “screw”. Additionally, for the “sidewalk” dataset, the attention response at defect locations of the model with randomly initialized weights are divergent. In contrast, the model under the MAML training strategy has stronger focusing ability on defect features and more accurate positioning accuracy.

CA effect

The CA module is embedded into the MeDetection model for further refinement of defect features. However, whether the location and number of CA modules have an impact on model performance is an important issue. Table 4 presents the simulation results, where three embedding ways are involved in the comparative experiments. In order to fully demonstrate the effect of CA location on model performance, the Average (Avg) and Variance (Var) statistical results are also presented in Table 4. We embed the CA module in the low- or high-level of the feature extraction channel, and also take into account the effect of both embedding. Embeddings ways show differences across all datasets. Generally, when the model is embedded with CA modules at low layers, it exhibits better performance than when embedded at higher layers. The performance of the model is not stable when CA modules are embedded in the low- or high-level of the feature extraction channel. For the datasets of “fabric2” and “fabric4”, embedding multiple CA modules improves the model performance. In addition, from the Average and Variance statistical results, we can obtain the similar conclusion, that embedding the CA mudule at low lever can help the model get the best performance with the maximum average accuracy and minimum variance.

Ablation experiment

Two feature refinement modules help the MeDetection model improve performance. The SN module further strengthens the defect features while unifying the defect representation. The CA module enhances the feature extraction ability of the model from the attention perspective. The ablation experiment is carried out to verify the roles of the two modules, and Table 5 shows the simulation results, where four datasets are taken into consideration. Obviously, the addition of the two modules greatly improves the performance of the MeDetection model. Specifically, after removing the SN module, the performance of the MeDetection model drops the most. For recognition accuracy (ACC), the performance of model without SN module drops by 18 percentage points for the dataset of “fabric4”. Undoubtedly, the performance of the model degrades the most when both the SN and CA modules are removed.