Computer vision-based algorithm for precise defect detection and classification in photovoltaic modules

Progressive annotation for photovoltaic module defect detection

PV modules are necessary devices with multiple battery cells that can convert solar energy into electrical energy. The PV modules inevitably experience material degradation or damage. These defects will reduce the efficiency of PV power generation. In severe cases, it may even cause the entire system to malfunction. Therefore, defect detection technology for PV modules is crucial for ensuring the regular operation and power generation efficiency of PV systems. Labeling the collected PV module images manually is time-consuming and labor-intensive. Therefore, the research adopts progressive annotation technology, which can quickly achieve high-precision annotation for a large amount of image data, significantly improving annotation efficiency and accuracy. The progressive annotation process is shown in Fig. 1.

Our experimental design was meticulously organized to guarantee reliability and correctness. The study maintained control variables, including consistent lighting conditions, uniform camera settings, and a controlled interior atmosphere. The data collecting technique entailed utilizing advanced cameras mounted on drones to take 10,000 high-resolution images of PV modules. Specialists manually analyzed a subset of 1,000 images while a progressive annotation process tagged the other images. The preprocessing stages encompassed enlarging the photos to dimensions of 512 × 512 pixels, leveling the pixel values, implementing data augmentation techniques such as horizontal flipping, rotation, and scaling, and removing noise by Gaussian blurring. In addition, segmentation masks were generated to outline defects accurately. These measures established a strong basis for assessing the effectiveness of our fault identification system, which relies on computer vision technology.

In Fig. 1, the image data that need to be annotated are sourced from the cruise acquisition of drones. In these data, various defects are included, such as yellowing of the back panel, dust, and snail marks. These defects are usually minor. Therefore, progressive annotation technology can accurately complete tasks such as image classification, object detection, and image segmentation (Pierdicca et al., 2020). After training with annotated image data, a segmentation model is obtained for inference in actual scenarios. When the model infers in actual scenarios, if the effect does not meet expectations, operations such as modifying, deleting, or supplementing the annotated data can be performed to train the model better. After multiple revisions and training, the ideal annotation can ultimately be obtained (Fang et al., 2022). Finally, by combining generate adversarial networks (GAN) data and defect sample data, a defect sample dataset can be constructed for subsequent research and application (Haque et al., 2019). The obtained defect sample dataset is shown in Table 1.

In Table 1, the data in the defect sample dataset comes from the annotated defect samples and the GAN automatically synthesized defect samples. GAN is one of the most promising methods for unsupervised learning on complex distributions. It consists of two modules: a generator and a discriminator (John, Suma & Athira, 2022). The generator is to generate fake data, which is similar to real data. The discriminator distinguishes between actual and fake data. These two modules confront each other, constantly adjusting parameters and generating high-quality and diverse fake data. The mathematical expression of GAN’s loss function is shown in Eq. (1). (1)${\displaystyle \underset{G}{min}\underset{D}{max}{L}_{\left(G,D\right)}=E_{z\sim P_z\left(z\right)}\left[log\left[1-D\left(G\left(z\right)\right)\right]\right]+E_{x\sim P_{date}\left(x\right)}\left[logD\left(x\right)\right]}$

In Eq. (1), ${min}_G{max}_D{L}_{\left(G,D\right)}$ represents the loss function of GAN. G and D represents generators and discriminators, respectively. x and z represent real data and random noise, respectively. $logD\left(x\right)$ represents the discriminator’s discrimination result on the data. In this log function, if the value D approaches 0 infinitely, the gradient decreases and the parameter updates to slow down (Sharma, Nair & Gomathi, 2022). To address this limitation, the loss function of GAN is adjusted to cross-entropy loss, as shown in Eq. (2). (2)${\displaystyle \underset{G}{min}\underset{D}{max}{L}_{\left(G,D\right)}=E_{z\sim P_z\left(z\right)}\left[logD\left(G\left(z\right)\right)\right]+E_{x\sim P_{date}\left(x\right)}\left[logD\left(x\right)\right]}$

Mask-RCNN is a deep-learning model mainly used for segmentation and mask prediction (Smarandache, 2022). This model combines object detection and semantic segmentation to accurately detect and segment multiple instances in an image and generate masks for each instance. Therefore, Mask-RCNN is used as a segmentation model for data training. Figure 2 displays the structure of Mask-RCNN.

In Fig. 2, the Mask-RCNN model mainly consists of a feature pyramid network (FPN), a region proposal network (RPN), a fully connected layer, and a region feature aggregation layer (ROIAIign). FPN is a network architecture used for object detection, which can fuse feature maps at different levels to obtain feature maps from low to high levels, capturing multi-scale contextual information (Lin et al., 2022). RPN is an object detection network architecture specifically designed to extract candidate boxes. This architecture can predict areas in the input image containing target objects. These candidate regions can provide a basis for subsequent feature extraction and classification. In the RPN model, each pixel generates multiple candidate boxes of different sizes. These candidate boxes are continuously modified through several parameters to make them closer to the target box. During this process, the model will select the candidate box with the highest confidence based on the confidence levels of the candidate boxes before and after the correction. Finally, all candidate boxes are screened using non-maximum suppression (NMS) to obtain the final target box. The mathematical expression for candidate box correction is shown in Eq. (3). (3)${\displaystyle \left\{\begin{array}{c}a:=\left(1+\Delta a\right)\cdot a\hfill \\ b:=\left(1+\Delta b\right)\cdot b\hfill \\ c:=exp\left(\Delta c\right)\cdot c\hfill \\ d:=exp\left(\Delta d\right)\cdot d\hfill \end{array}\right.}$

In Eq. (3), $\left(a,b\right)$ and $\left(c,d\right)$ stand for the candidate box’s center point coordinates and dimensions. a:, b:, c: and d: represent the corrected size. Δa, Δb, Δc and Δd represent the size obtained during the candidate box correction process. The Mask-RCNN model includes object detection tasks in structure and adds a Mask branch for pixel classification (Lotfollahi et al., 2022). Therefore, when calculating the loss function, the loss function generated on the Mask branch is calculated. The mathematical expression of the loss function is shown in Eq. (4). (4)${\displaystyle L=L_{cls}+L_{box}+L_{mask}.}$

In Eq. (4), L represents the loss function. L_cls represents the loss function generated by the segmentation model during object classification. L_box represents the loss function generated by the segmentation model when generating candidate boxes. L_mask stands for the loss function generated by the segmentation model on the Mask branch. The Mask branch of the Mask-RCNN model can have binary pixels; each pixel is classified as a foreground object or background. This process can greatly avoid competition between categories, improving instance segmentation accuracy.

Transfer learning for PV module defect detection

ROIAlign is a regional feature aggregation method in Mask R-CNN. This method solves the region mismatch caused by two quantization operations in the region of interest pooling. Its main function is to convert feature maps of any size region of interest into small feature maps with fixed sizes, thereby improving recognition accuracy. The principle of the ROIAlign layer is shown in Fig. 3.

In Fig. 3, the ROIAlign layer can accurately map the target box onto the feature map for pooling operations. It meticulously calculates the pixel features covered by the pooling kernel through bilinear interpolation, significantly reducing the omission of feature information. The loss function measures the difference between predicted and actual results, which plays a crucial role in machine learning and optimization problems. To improve the prediction accuracy of Mask-RCNN, the loss function setting of the model is optimized. The classification function is mainly used to distinguish between the background and the target object. The mathematical expression of the optimized classification loss function is displayed in Eq. (5) to improve the confidence level of retaining candidate box positions. (5)${\displaystyle L_{cls}^{\prime }=\frac{1}{{\alpha }_{cls}}\sum -log\left[\underset{i}{\overset{\ast }{P}}{P}_i+\left(1-P_i\right)\left(1-P_i\right)\right]}$

In Eq. (5), $L_{cls}^{\prime }$ represents the optimization classification loss function. α_cls represents the normalization parameter. P_i represents the probability of positive sample output for the ROI of the ith ordinal. $P_i^{\ast }$ represents the adjustment parameter. When the candidate region is a positive sample, $P_i^{\ast }=1$. When it is a negative sample, $P_i^{\ast }=0$. To make the coordinate center of the candidate box closer to the coordinate center of the real object in the image, the optimized L_box function expression is shown in Eq. (6). (6)${\displaystyle L_{box}^{\prime }=\frac{1}{{\alpha }_{box}}\sum \underset{i}{\overset{\ast }{P}}R\left(t_i,t_i^v\right)}$

In Eq. (6), $L_{box}^{\prime }$ represents the optimized L_box function. α_box represents the normalization parameter. $t_i^v$ and t_i represent actual and predicted migration parameters, respectively. R represents the SmoothL1 loss function, as shown in Eq. (7). (7)${\displaystyle Smoot{h}_{L1}=\left\{\begin{array}{c}0.5X^{2}if\left|X\right|<1\hfill \\ \left|X\right|-0.5otherwise\hfill \end{array}\right.}$

The optimized mask loss function is shown in Eq. (8) to improve the classification accuracy of candidate boxes. (8)${\displaystyle L_{mask}^{\prime }=\frac{1}{{\beta }^2}\left[\sum y_{v}log{y}_v^k+\left(1-y_v\right)log\left(1-y_v^k\right)\right]}$

In Eq. (8), $L_{mask}^{\prime }$ represents the optimization mask loss function. β represents the output size of the feature map. $y_v^k$ and y_v represent predicted label values and true label values, respectively. The output process of fixed size feature maps is shown in Fig. 4.

In Fig. 4 W and H stand for the width and height of the convolutional kernel. After processing, five feature maps of different sizes are obtained. Transfer learning is the strategic methodology of repurposing knowledge, models, or techniques acquired in one domain to enhance problem-solving in a different yet related domain. This process hinges on the commonalities or connections that exist between various fields. Such an approach allows the transfer of pre-existing competencies to novel contexts and applications, effectively expediting the learning curve and bolstering performance outcomes. Within machine learning, transfer learning is mainly focused on the challenge of domain adaptation—how to effectively deploy a model trained in one task or domain to a new, albeit related task, ensuring it maintains robust performance on unfamiliar endeavors.

The limitations of transfer learning are shown in Eq. (9). (9)${\displaystyle \left\{\begin{array}{c}P\left(X_S,Y_S\right)\ne P\left(X_T,Y_T\right)\hfill \\ P\left(X_S\right)\ne P\left(X_T\right)\hfill \\ P\left(X_S\left|Y_S\right.\right)\ne P\left(X_T\left|Y_T\right.\right)\hfill \end{array}\right.}$ In Eq. (9), $P\left(X_S,Y_S\right)$ and $P\left(X_T,Y_T\right)$ represents the marginal probability of the source domain and the marginal probability of the target domain, respectively. In transfer learning, pre-trained models can be used as initial points. Then, the model parameters can be fine-tuned to adapt to the new dataset. When using the Mask-RCNN model for training, truncated normal distribution can be used to initialize model parameters to ensure that the coordinate center of the candidate box is closer to the coordinate center of the real object in the image. The probability density function of the normal distribution is shown in Eq. (10). (10)${\displaystyle \phi \left(\zeta \right)=\frac{1}{\sqrt{2\pi }}exp\left(-\frac{1}{2}{\zeta }^2\right)}$

In Eq. (10), $\phi \left(\zeta \right)$ represents the probability density function. The cumulative distribution function is shown in Eq. (11). (11)${\displaystyle \varphi \left(\varsigma \right)=\frac{1}{2}\left(1+erf\left(\varsigma /\sqrt{2}\right)\right)}$

In Eq. (11), ς represents the parameter of the normal distribution. When the limit of X is set to $X\in \left(a,b\right),-\infty \le a<b\le \infty$, the probability density function of X is shown in Eq. (12). (12)${\displaystyle f\left(X;\mu ,\sigma ,a,b\right)=\left\{\begin{array}{c}\frac{\phi \left(\frac{x-\mu }{\sigma }\right)}{\sigma \left(\phi \left(\frac{b-\mu }{\sigma }\right)-\phi \left(\frac{a-\mu }{\sigma }\right)\right)},a<X<b\hfill \\ 0,otherwise\hfill \end{array}\right.}$

In Eq. (12), $\left(\mu ,{\sigma }^2\right)$ represents the limiting parameter. To verify the learning effectiveness, the core evaluation indicator of intersection and union ratio can be used to measure the accuracy and comprehensiveness. The definition for precision is shown in Eq. (13). (13)${\displaystyle P=\frac{TP}{FP+TP}}$

In Eq. (13), TP and FP represent true and false positive examples, respectively. The definition of recall rate is shown in Eq. (14). (14)${\displaystyle R=\frac{TP}{FN+TP}}$ In Eq. (14), FN represents a false counterexample. The source domain data used for transfer learning comes from ImageNet. The data in the target domain is the defect sample dataset. In transfer learning, pre-trained models are often used as the foundation. The parameters are fine-tuned to adapt to new datasets.

In some cases, a mechanism called auxiliary loss can be introduced to solve the gradient disappearance caused by a too deep network. This auxiliary loss can enhance the expressive ability and facilitate gradient backpropagation, thereby improving the convergence speed and performance of the model. The adjustment coefficient of auxiliary loss is shown in Eq. (15). (15)${\displaystyle \gamma =\left\{\begin{array}{c}0.1\ast 2^i,0\le i\le 2\hfill \\ 10^i,3\le i\le -1\hfill \\ 0,otherwise\hfill \end{array}\right.}$

To verify the effectiveness of the defect detection and classification algorithm for PV modules based on computer vision technology, the experimental environment configuration is shown in Table 2. CUDA is the computing platform, and Python is the programming language for progressive automatic annotation. The number of progressive self-training iterations is 2,000. The learning rate is 0.0001. The number of training steps is 200. The weight attenuation coefficient is 0.0001. The maximum and minimum dimensions of the sample set are set to 512.

We employed paired t-tests with a significance level of 0.05 to evaluate the null hypothesis that there is no significant difference in performance between our method and existing methods. Our methodology yielded statistically significant enhancements. In addition, we employed the bootstrap method to determine 95% confidence intervals for the performance metrics, which indicates the range within which the true metrics are anticipated to fall with 95% confidence. These analyses rigorously validated the effectiveness of our computer vision-based methodology.

To improve the clarity and comprehension of our approach, we offer thorough elucidations and rationales for the fundamental equations employed in our defect detection model. The GAN loss function is essential for producing authentic synthetic data, augmenting the training dataset and enhancing the model’s resilience. The modified GAN loss function effectively tackles the issue of the vanishing gradient problem, hence guaranteeing a stable training process. The RPN candidate, box correction equation, guarantees precise proposal generation, improving object detection accuracy. The Mask-RCNN loss function integrates classification, bounding box regression, and mask prediction losses to achieve thorough defect identification and segmentation. The optimized classification loss function effectively equalizes the influence of positive and negative data, mitigating the risk of overfitting. The Smooth L1 loss function, employed in bounding box regression, achieves a trade-off between sensitivity to outliers and convergence speed. Utilizing the normal distribution function for parameter initialization facilitates expedited convergence and enhanced performance. The IoU metric and recall rate are crucial for assessing the accuracy of a model and guaranteeing dependable fault identification. Together, these equations enhance our defect detection system’s resilience, precision, and efficiency, guaranteeing its practical and theoretical efficacy in photovoltaic module quality control.

Performance verification of Mask-RCNN model

To assess the efficacy of Mask-RCNN, the model is fed with preprocessed imagery. Post iterative self-improvement training, the resulting loss curve and accuracy metrics are gleaned, as depicted in Fig. 5. From Fig. 5A, the loss curve of the Mask-RCNN model gradually stabilized after 600 iterations. There were some fluctuations between 1,000 and 1,500 iterations. Overall, the general trend of the loss curve still showed a convergence trend, but the stability was insufficient. From Fig. 5B, the training accuracy of the Mask-RCNN model reached 90.8% when the iterations were 400. Then it fluctuated around 90%. When the number of iterations approached 1,250, there was a significant decrease in accuracy, possibly due to the model encountering images that had not been learned before but then stabilizing around 90%.

After completing the model training, a test set containing 100 samples was selected to test the model. The classification accuracy in different defect sample categories was obtained, as shown in Fig. 6. From Fig. 6, the classification accuracy of all defect sample categories was higher than 85%. The classification accuracy of yellowing on the back plate was the highest at 89.2%, while the accuracy of dust was the lowest at 85.3%. However, by comparing the model accuracy on the training set, the model accuracy on the test set has not reached the original level. This may be due to the trained model reaching a saturation state of accuracy. A common approach was to introduce transfer learning to improve classification accuracy further.

Mask-RCNN model validation based on transfer learning training

To verify the performance of the Mask-RCNN model after transfer learning, 25%, 50%, and 100% data were used for training. After training, the trained model was tested on the test set. Table 3 displays the test accuracy results. From Table 3, the Mask-RCNN model after transfer learning exhibited excellent performance when trained with 100% data. The minimum accuracy reached 97.3%, while the highest accuracy was 99.1%. This result showed that transfer learning can significantly improve classification accuracy when the training data is sufficient. When trained with 50% of the data, the model’s testing accuracy remained excellent, exceeding 90%. This indicated that although the training data was reduced by half, the model could still effectively learn and recognize the target object. However, when training with 25% data, the accuracy performance of the model was not ideal. Because of a significant reduction in the training data, the model could not thoroughly learn and master all classification features. In summary, from the experimental results, the ideal classification accuracy could be obtained when the training data volume was greater than 50%. On this basis, increasing the training data will further improve the accuracy.

To investigate the performance further, the value of the auxiliary loss adjustment coefficient was set $\left[0,0.01,0.1,0.2,0.4\right]$. The impact of different adjustment coefficients on model accuracy is shown in Fig. 7. From Fig. 7, the highest accuracy on the training set was 97.5%, and the highest accuracy on the test set was 96.8%. After adding the auxiliary loss adjustment coefficient γ, the accuracy did not significantly improve. Meanwhile, the size of the adjustment coefficient γ, the higher the accuracy, but without it, the accuracy would decrease. This is because the main function of γ isto jump connect, which transmits information from the middle of the structure to the deep part of the structure without loss.

To more intuitively verify the effectiveness of this research model, the Mask-RCNN model was compared with the following four different defect detection methods: ITV, Gabor, AD, and LT methods. The performance comparison of five defect detection methods is shown in Fig. 8. From Fig. 8A, the accuracy of the Mask-RCNN model was 98.7%. Compared with ITV, Gabor, AD, and LT methods, it has increased by 14.1%, 17.2%, 3.6%, and 5.9%, respectively. From Fig. 8B, the recall rate of the Mask-RCNN model was 0.913. Compared with the ITV, Gabor, AD, and LT methods, it has increased by 19.5%, 11.89%, 6.04%, and 29.13%, respectively. Overall, the Mask-RCNN model had better detection performance.

To comprehensively verify the superiority of this research model, the efficiency of these five different defect detection methods was compared, as shown in Table 4. From Table 4, the inference time of the Mask-RCNN model was 3.53 ms, and the detection speed was 280.69 fps. Compared to the current methods, our algorithm significantly enhances the detection of defects in photovoltaic modules, with quantitative justifications that emphasize these improvements. The efficiency was significantly better than the other four defect detection methods. Compared with the ITV, Gabor, AD, and LT methods, their inference time decreased by 76.84%, 78.50%, 96.90%, and 97.25%, respectively. The detection speed has increased by 4.22 times, 4.49 times, 32.68 times, and 35.73 times, respectively. These results demonstrated that the Mask-RCNN model has significant advantages in detection speed.

Automated quality control in PV manufacturing

The implementation of our computer vision-based algorithm can significantly improve the accuracy of defect detection during the manufacturing process. By promptly detecting defects such as micro-cracks, delamination, and foreign object occlusions, manufacturers can prevent defective modules from reaching the market, thereby ensuring the quality of their products. The dependence on manual inspection, which is labor-intensive and susceptible to human error, is diminished by automated defect detection. This can result in substantial cost savings in labor costs and reduced discard rates, as defective modules can be identified and rectified before their final assembly.

Efficiency improvements in energy production

Ensuring that PV modules are devoid of flaws and functioning at maximum efficiency directly impacts the energy production of PV systems. Our methodology may effectively preserve PV installations’ peak power generation capability by rapidly resolving any faults that may otherwise lead to performance degradation. Malfunctions in PV modules can result in substantial reductions in energy output over some time. Our technique can successfully identify and address these flaws, resulting in decreased energy wastage and improved efficiency of PV systems.

Economic and environmental impact

PV systems’ enhanced efficacy and decreased operational interruptions contribute to more economically efficient energy generation. By reducing the levelized cost of electricity (LCOE), solar power can become more economically viable and competitive than conventional energy sources. Our flaw detection system promotes the renewable energy sector’s sustainability objectives by prolonging the PV modules’ operational lifespan and optimizing their performance. This helps decrease the environmental footprint linked to the manufacturing and disposal of faulty modules.

Ethical and environmental considerations

Deploying our flaw detection technique in large-scale PV operations requires us to tackle many ethical and environmental concerns. From an ethical standpoint, it is of utmost importance to guarantee the confidentiality and protection of data while also offering chances for retraining workers impacted by automation. From an environmental perspective, our approach improves resource efficiency by minimizing material waste and increasing the lifespan of PV modules. Nevertheless, it is crucial to enhance the energy efficiency of AI systems and implement strong e-waste management policies to reduce their environmental footprint. Through strict compliance with environmental rules and thorough lifecycle studies, we can guarantee that our technology actively encourages sustainable practices within the PV industry.

Integration of advanced AI techniques

Utilization of explainable AI (XAI)

Real-time monitoring and predictive maintenance

Multimodal data integration