Defect text recognition and condition assessment method of power equipment based on BERT-BiLSTM

Overview of power equipment defect texts

Defect records of power equipment are mainly generated by operation and maintenance personnel, who enter defect information from daily inspections, maintenance, and equipment operation into the Power Management System (PMS) following a certain template (Wang, Cao & Lin, 2019). These records include objective and subjective details such as equipment type, defect location, date, description, severity, causes, and clearance time, each corresponding to a specific defect event of a transformer (Ye, Guo & Qi, 2023; Bhattarai et al., 2019).

Because defect texts are completed by personnel with varying experience, their formats are diverse, descriptions may be vague or oversimplified, and occasional errors exist in severity or component labels. Nevertheless, defect descriptions often contain critical information on defect occurrence, location, and phenomena, serving as an important supplement to other record fields—especially for equipment defects with unclear boundaries.

In practical grid operations, defect severity, typically classified as general, serious, or critical, is manually assigned, making the process labor-intensive, subjective, and inconsistent. However, severity classification is crucial for fault management, directly influencing required response times and maintenance prioritization. With advancements in natural language processing, deep learning techniques applied to historical defect text data enable automated extraction of key semantic features, achieving rapid and accurate classification. This approach reduces manual workload and provides a robust foundation for maintenance planning, fault diagnosis, and transformer health assessment (Zhang et al., 2021; Wang et al., 2019; Liao et al., 2021; Dong et al., 2019; Tamma, Prasojo & Suwarno, 2021; Zhang et al., 2023).

Analysis of the characteristics of defect texts of power equipment

Partial defect text records are shown in Table 1. Compared with general Chinese text, transformer defect records are highly variable due to manual entry and contain strong subjectivity. Their main characteristics are as follows:

• (1)

  Defect text belongs to a specialized domain of the power system and includes numerous technical terms. Influenced by personnel experience, professional level, and language habits, the same equipment may be described differently, such as “main transformer” and “main power transformer”, “oil storage tank” and “oil pillow”.

• (2)

  Significant variation in length, from a few characters to over 100, caused by differences in inspection methods, scope, and language style.

• (3)

  Frequent inclusion of letters, numbers, abbreviations, and formulas. Quantitative details, such as “discoloration exceeds 2/3 of the total amount”, often determine defect severity, adding to mining difficulty.

• (4)

  Semantic loss or redundancy from limited operator experience, such as missing severity levels, vague locations, or irrelevant details, which introduce noise and reduce mining accuracy.

Theory of BERT model

The BERT model, proposed by Google in 2018, employs a bidirectional Transformer encoder and unsupervised masked language modeling on large corpora to enhance natural language understanding and bidirectional representations (Devlin et al., 2018; Vaswani et al., 2017; Yang & Jiang, 2018). Unlike RNNs and CNNs, BERT’s attention mechanism reduces token distance to one, effectively handling long-range dependencies. For transformer defect texts, BERT accurately extracts key information such as defect type, component, and severity. Its structure is shown in Fig. 1.

In Fig. 1A, the BERT feature extractor consists of four layers. The first layer (E₁–E_n) is the embedding layer, combining token and positional embeddings to preserve semantics and order. The second and third layers are stacked Transformer encoder blocks (“Trm”) using self-attention to capture multi-level contextual dependencies for deep semantic modeling of defect texts, their structure is shown in Fig. 1B. The final layer (T₁–T_n) outputs contextualized token vectors for downstream BiLSTM sequence modeling.

In Fig. 1B, the Transformer core includes a self-attention mechanism and a Feedforward Neural Network (FNN), used to extract defect text vector features. BERT’s self-attention models relationships between any two tokens, capturing semantics and internal structure. The self-attention output is computed as:

(1) $$Attention\left(Q,K,V\right)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$where Q, K, and V are query, key, and value matrices, and d_k is their dimension.

Compared with a single self-attention mechanism, multi-head attention extends this by projecting into n subspaces with distinct $W_i^Q$, $W_i^K$ and $W_i^V$, enabling the model to focus on multiple aspects simultaneously:

(2) $$MultiHead\left(Q,K,V\right)=Concat\left(h_1,\cdots ,h_i,\cdots h_n\right)W^o$$

(3) $$h_i=Attention\left(Q{W}_i^Q,K{W}_i^K,V{W}_i^V\right)$$where Concat represents the concatenation of all h_i matrices obtained from the i-th self-attention mechanism, followed by matrix multiplication with the concatenation matrix; n is the heads in the multi-head self-attention mechanism, typically set to 8; $W_i^Q$, $W_i^K$ and $W_i^V$ are the weight matrices of matrices Q, K, V in the i-th subspace.

Residual connections and layer normalization are applied after self-attention to preserve gradients, stabilize training, and improve convergence (Chen et al., 2020). The normalized output passes through an FNN with two fully connected layers, followed by another residual and normalization step, and this process is repeated. The output of the FNN model is calculated as follows:

(4) $$x_{FNN}=ReLU\left(0,x\cdot w_1+b_1\right)\cdot w_2+b_2$$where x is the input to the feedforward neural network; w₁, b₁ and w₂, b₂ are the weights and biases of the two fully connected layers, respectively.

Theory of BiLSTM model

LSTM neural network

The LSTM neural network, proposed by Hochreiter and Schmidhuber (Jang et al., 2020; Zhou et al., 2019), enhances long-term dependency modeling by introducing three gates (input i, forget f, and output o) to control information flow within a memory cell. LSTM adapts to variable-length sequences and maintains contextual consistency, making it suitable for tasks such as Natural Language Processing (NLP), speech recognition, and time-series forecasting. Its internal structure, shown in Fig. 2A, comprises the three gates and a memory cell.

The forget gate determines how much of the previous cell state is retained or discarded. As shown in Eq. (5), it processes the previous hidden state h_t-1 and current input x_t, outputting a value between 0 (complete discard) and 1 (full retention):

(5) $$f_t=\sigma \left(w_f\cdot \left[h_{t-1},x_t\right]+b_f\right)$$where f_t is the forget gate state; w_f and b_f are its weights and bias, and σ is the sigmoid activation function.

The input gate regulates new information entering the cell state. It evaluates h_t-1 and x_t, to decide which information to store, with a tanh layer generating candidate memory values. The updated cell state C_t combines retained and new information, as expressed in Eq. (6).

(6) $$i_t=\sigma \left(w_i{h}_{t-1}+w_i{x}_t+b_i\right)$$

(7) $$c_t=\tan h(w_c{h}_{t-1}+w_i{x}_t+b_c)$$

(8) $$C_t=C_{t-1}\otimes f_t+i_t\otimes c_t$$where i_t is the input gate state at time step t, controlling the amount of information from x_t passed to C_t; w_i and b_i are the weights and biases of the input gate respectively; w_c and b_c are the weight matrix and bias term of the cell state; tanh is the hyperbolic tangent activation function, and $\otimes$ denotes the Hadamard product.

The output gate determines which part of the cell state is exposed as the hidden state. A sigmoid layer selects relevant cell state elements, which are then scaled by tanh to produce the output h_t, also serving as the next hidden state input. The output gate is calculated as follows:

(9) $$o_t=\sigma \left(w_o{h_t}_{-1}+w_o{x}_t+b_o\right)$$

(10) $$h_t=o_t\otimes \tan h(C_t)$$where o_t is the output gate state at time step t; h_t-1 is the previous time step’s hidden state; w_o and b_o are the weight matrix and bias of the output gate, respectively.

Defect text recognition model

To effectively extract important features from defect texts and accurately identify defect characteristics such as performance, component, and severity in the defect texts, a defect text classification model based on BERT-BiLSTM is established, as shown in Fig. 3.

First, the word embedding layer converts the input sequence into word vectors using the pre-trained BERT model. In Fig. 3, Tok1-TokN correspond to the characters or words in the defect text. The [CLS] token added at the beginning of the text represents the vector T_CLS obtained after training the BERT model, which can be used for subsequent text classification. Contextual word vectors are then generated by BERT as input to the BiLSTM. Here, the Chinese BERT-Base model is used, comprising 12 transformer layers with hidden size 768, pre-trained on large-scale Chinese corpora and fine-tuned on the cleaned dataset to capture defect-domain linguistic patterns. The BiLSTM encodes forward and backward context, capturing long-range dependencies in the BERT output sequence. Finally, the decoding layer maps BiLSTM outputs to classification space via fully connected layers, and the Softmax classifier predicts defect labels.

Transformer defects text dataset setup

Based on 10,550 Alternating Current (AC) transformer defect records above 35 kV from a provincial power grid over 5 years, a defect text dataset was constructed through manual cleaning and screening. Original records, written by personnel with varying expertise, included inconsistent terminology (e.g., “main transformer” vs. “power transformer”), vague expressions (e.g., “abnormal noise” without location), and inaccurate component labels. Cleaning involved standardizing terms (e.g., unifying “cooling fan” variants), resolving ambiguities (e.g., cross-referencing components), and verifying labels (e.g., correcting severity ratings from context). After removing duplicates, errors, and invalid entries, and manually correcting missing information, 2,464 text records remained, split into training and testing sets at a 3:1 ratio. Classification labels, defect performance, component, and severity, were assigned based on the dataset and standards. The label categories and their meanings are listed in Table 2.

The labels of each defect text in the defect text dataset are determined. The distribution of each label category in the defect dataset is shown in Fig. 4.

In the three label categories, sample distribution varies notably for defect severity. This due to the substantial differences in the original defect records. During routine inspections, most defects are classified as General or Serious, with few Critical cases due to low risk of major accidents. For defect components, the top five classes, tanks, cooling systems, on-load tap-changers, bushings, and non-electrical protection, have similar record counts, while defects in iron cores and windings are rare, as they are harder to detect. Regarding defect types, mechanical, insulation, and thermal defects have comparable sample sizes, whereas other types are scarce (Li & Li, 2017), consistent with expert assessments.

Classification result analysis of the BERT-BiLSTM model

This model is based on the Windows operating system and uses Python 3.7 programming language. The testing and analysis are conducted based on the TensorFlow deep learning framework and the Keras neural network framework. The key parameters of the model are selected, as shown in Table 3.

Hyperparameters were determined via empirical tuning and transformer text classification studies. A learning rate of 5 × 10⁻⁵ and batch size of 128 ensured stable convergence. To mitigate overfitting on the moderate, noisy dataset, a dropout layer (0.5) followed the BiLSTM, with early stopping applied based on validation loss. Hidden layer size balanced expressiveness and computation, and 30 epochs sufficed for convergence. Rectified Linear Unit (ReLU) activation and Adam optimizer were used, with Adam adaptively minimizing loss.

To address class imbalance, class weights—set inversely proportional to class frequency—were incorporated into the loss function,increasing the gradient impact of minority classes and reducing bias toward majority classes without resampling or architectural changes.

Effective text classification is an important module for assessing the performance of text recognition models. In this article, model performance is evaluated using precision, recall, F1-score, accuracy, and the confusion matrix, defined as follows:

(11) $$Accuracy={\displaystyle \frac{T_P+T_N}{T_P+T_N+F_P+F_N}}$$

(12) $$Precision={\displaystyle \frac{T_P}{T_P+F_P}}$$

(13) $$Recall={\displaystyle \frac{T_P}{T_P+F_P}}$$

(14) $$F1\text{-}score={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}$$where T_P, T_N, F_P, and F_N correspond to the numbers of true positives case, true negatives case, false positives case, and false negatives case, respectively.

Taking defect severity classification as an example, Fig. 5 presents the model’s performance on the training and testing sets. Although fluctuations occur during training, accuracy and loss gradually converge with increasing epochs, stabilizing around epoch 18. At 30 epochs, the model achieves an accuracy of 96.83%.

The evaluation results of the BERT-BiLSTM model for the three defect text classification tasks are shown in Table 4. The model achieves over 94% accuracy for defect severity and defect component classification, while defect type classification attains a comparatively lower accuracy of over 84%.

The confusion matrix results for the three types of text classification problems are shown in Fig. 6. The BERT-BiLSTM model has good performance in defect severity and component classification, but low in defect type recognition.

Misclassification of defect severity mainly arises between general and severe levels due to vague descriptions. Component errors frequently occur among tank, conservator, and non-electrical protection devices when multiple components are mentioned ambiguously. Defect type classification is hindered by sample imbalance and fuzzy semantic boundaries, particularly between mechanical and thermal faults. Many “other” defects, such as auxiliary system issues, share textual features with overheating or insulation faults, leading to misclassification. Moreover, independent category prediction in the BERT-BiLSTM model neglects correlations between defect type and component, reducing classification accuracy.

Comparison with general classification models

Based on the above evaluation metrics, the effectiveness of the BERT-BiLSTM model and commonly used text classification models such as CNN, BERT, and LSTM are compared, as shown in Fig. 7.

The results show that the evaluation metric distribution across the three classification tasks is consistent with other models, but BERT-BiLSTM achieves the highest F1-score (>84%), 2–5% higher than alternatives, indicating superior semantic extraction for transformer defect texts. The F1-score ranking is BERT-BiLSTM > BERT > LSTM > CNN. The reasons may be: (1) LSTM alone outperforms BERT, suggesting that while LSTM efficiently mines defect text features, its global feature extraction contains noise that BERT’s bidirectional encoding can mitigate. (2) CNN performs worst, particularly with long texts, due to its shallow sequential feature extraction and inability to capture deep semantic context.

In addition to classification performance, the computational efficiency and resource demands of the four models are shown in Table 5. The BERT-BiLSTM model has the longest training time (1 h 13 min for 30 epochs), the largest parameter count (~113.5 M), and peak GPU usage (~6.3 GB) due to its dual-layer architecture. Once trained, inference is efficient, taking only a few hundred milliseconds per sample. By contrast, CNN and LSTM consume fewer resources but capture complex semantic dependencies less effectively, leading to lower classification accuracy, highlighting the advantage of BERT-BiLSTM for tasks requiring precise diagnostics.

Transformer condition assessment

Transformer condition assessment, fundamental for equipment maintenance decision-making, evaluates equipment health based on operation, maintenance, and testing. Condition maintenance uses these assessments to determine operational status and guide maintenance plans. Results are expressed as condition scores (quantitative) and grades (qualitative), which correspond to each other, though grade assignment may vary across standards or procedures.

In transformer health management systems, condition assessment often depends on online monitoring data, offline diagnostic test results, and analysis of feature quantities such as dissolved gases to evaluate the health status of the transformer. However, the influence of historical defect events (recorded as unstructured textual descriptions) on equipment condition should not be neglected. Despite their availability, these records are rarelyu sed as direct inputs of health assessment models.

To address this gap, an assessment method that integrates defect text recognition with transformer condition evaluation is proposed, as shown in Fig. 8.

This method first uses a BERT BiLSTM classification model to extract defect-related information (such as defect type, component, and severity). Then, according to the national transformer health assessment standards, these features are mapped to structured evaluation criteria, enabling automatic generation of condition scores and grades based on defect history. This framework cannot only broaden the data sources used in condition evaluation, but also support intelligent fault triage and maintenance decision-making in scenarios where sensor data may be unavailable or insufficient.

Association rules for status evaluation

In actual work, based on the understanding of equipment principles and operating conditions, a reasonable equipment defect classification method can be formulated. Considering the internal structure of the equipment, external environment, and operating conditions, combined with the actual status of the equipment, the defect rating score for power transformer and the weighting of defect status score are shown in Table 6.

In this framework, the BERT-BiLSTM model’s classification outputs—defect severity, component, and performance—serve as inputs to the scoring system. Predicted severity levels are mapped to score deductions (Table 5), and component types are assigned weights (Table 6). Label definitions and scoring thresholds follow national standards (e.g., DL/T 572, Q/GDW 11247) and are validated by maintenance experts.

Component weights reflect criticality and operational risk, determined via expert judgment and historical fault analysis. An adaptive correction mechanism dynamically adjusts health scores based on abnormal operation indicators, improving contextual accuracy. Expert validation indicates this approach better reflects operational risks than fixed weights, although formal ablation studies were not conducted.

According to the above condition evaluation rules, the deduction points of its transformer equipment are obtained, corresponding to the health status. Maintenance strategies can be developed based on the results of health index. The health index is calculated as follows:

(15) $$n_i=a_0{k}_0+a_1{k}_1+a_2{k}_2+a_3{k}_3$$

(16) $$HI={\displaystyle \frac{{\sum }_{i=1}^m{w}_i\times n_i}{{\sum }_{i=1}^{m}nmax{w}_i}}$$where a and k are respectively the number and score of different defect classes; n_i represents the overall score of the component; w_i represents the weight of the component, as shown in Table 6, and m represents the total number of classifications of defect component.

In traditional state evaluation, indicator weights remain fixed once determined. However, when an indicator exceeds normal values, its abnormal information may be underestimated due to a small original weight, which is impractical. To address this, a dynamic weight calculation with adaptive correction is employed. When an indicator shows a high state value, its importance in accurately assessing the current equipment health increases, making it a key factor in the comprehensive evaluation. Its weight is appropriately adjusted to reflect its critical impact on transformer condition. When n indicators exceed abnormal thresholds, they are recorded as set A, while m normal indicators form set B. The weights of the abnormal indicators are then adjusted as follows:

(17) $$w_i^{\mathrm{\prime }}=w_i+\left(1-{\sum }_{k\in A}{w}_k\right)w_{i}i\in A.$$

The remaining normal indicators are adjusted as follows:

(18) $$w_j^{\mathrm{\prime }}=w_j-\left(1-{\sum }_{k\in B}{w}_k\right)w_{j}j\in B$$

(19) $${\sum }_{i\in A}{w}_i^{\mathrm{\prime }}+{\sum }_{j\in B}{w}_j^{\mathrm{\prime }}=1.$$

Using the above scoring rules, the impact of historical defects on a transformer can be assessed to determine its health status and guide appropriate maintenance strategies. According to oil-immersed transformer condition evaluation guidelines, transformer status is categorized into four levels: good, fair, poor, and very poor. The corresponding maintenance programs for each status are listed in Table 6.

Case study

To verify the applicability of the proposed method for transformer condition assessment, we selected a real case involving a 35 kV AC transformer with multiple recorded defect events in the past year. The specific defect instances and their corresponding classification results are shown in Table 7.

Each defect text in Table 7 was processed by the BERT-BiLSTM model to extract three core elements: defect severity, component, and type. Component weights (w_i) were assigned based on a predefined expert knowledge base, and severity levels (k_i) were quantified via mapping rules. The overall Health Index (HI) was calculated using Eqs. (16) and (17), with a weighted aggregation across components. An HI of 0.44 corresponds to the “Fair” condition zone in Table 6, indicating moderate degradation. Although a critical defect (e.g., oil tank smoke) exists, its impact is offset by less severe defects and component weights. The recommended maintenance strategy is to shorten inspection intervals and perform targeted fault tracing, particularly for the tank and cooling system.

This case demonstrates the feasibility and effectiveness of the proposed framework and highlights the potential of NLP-based methods in digital power equipment management.