Research on license plate recognition based on graphically supervised signal-assisted training

Mainstream LPR architectures

Early deep learning approaches often adopted end-to-end recognition models. Seminal works like CRNN (Shi, Bai & Yao, 2016) combined CNNs for feature extraction with RNNs for sequence modeling, typically trained using CTC Loss (Graves et al., 2006) to handle variable-length character sequences without explicit segmentation. Addressing the computational overhead of RNNs, LPRNet (Zherzdev & Gruzdev, 2018) introduced a purely convolutional architecture for efficient, real-time LPR, also leveraging CTC Loss. Other approaches follow a two-stage pipeline, first detecting the license plate region and then recognizing the characters within it, sometimes employing models like Faster R-CNN for detection followed by specialized recognition networks (e.g., Vargoorani & Suen, 2024).

Strategies for improving LPR performance in challenging conditions

A primary thrust in LPR research is enhancing robustness against real-world image degradations.

Image restoration and super-resolution: Techniques aimed at improving input image quality are common. For instance, Nascimento et al. (2024) focused on layout-aware super-resolution, while Kim, Kim & Park (2024) developed AFA-Net for joint deblurring and super-resolution. Generative adversarial networks (GANs) have also been employed for super-resolution tasks in LPR (Zhang, Liu & Ma, 2018; Min, Lim & Gwak, 2020). Lee et al. (2019) integrated denoising and rectification directly within their SNIDER framework. These methods typically add preprocessing steps or incorporate complex restoration modules.

Attention mechanisms and architectural enhancements: Attention mechanisms have been widely adopted to help models focus on salient character regions (Wang & Chen, 2024; Zhang et al., 2020; Xu et al., 2021). Some works also explore novel architectures, such as the use of Vision Transformers (Ceng & Ke, 2021) or, more recently, visual language models like VehiclePaliGemma (AlDahoul et al., 2024), for tackling difficult recognition tasks.

Our approach differs from these works. Instead of modifying the input image or significantly altering the inference architecture of a lightweight model like LPRNet, we focus on enriching its learned representations through an auxiliary training signal derived from existing labels.

Leveraging auxiliary information and training paradigms

The concept of using additional information or tasks to aid model training is well-established. Font characteristics have been identified as an influential factor in LPR performance (Vargoorani & Suen, 2024).

In broader computer vision, self-supervised learning (SSL) methods excel at learning representations from unlabeled data by designing pretext tasks (e.g., He et al., 2020; Chen et al., 2020; He et al., 2022). While SSL typically derives supervision from the input data itself, our work is inspired by the principle of mining richer supervisory signals.

Our proposed graphically supervised signals (GSS) can be viewed within the context of auxiliary learning. We introduce a secondary task—reconstructing an image representation of the text label—to run in parallel with the primary LPR task. The key distinction of our GSS approach is the direct transformation of textual labels into a graphical (image) modality to provide explicit, pixel-level guidance. This contrasts with methods that might use abstract intermediate representations or different types of auxiliary losses not directly tied to the visual rendering of the target text.

Focus on lightweight model enhancement

The practical demand for efficient LPR systems necessitates a focus on lightweight models (Zherzdev & Gruzdev, 2018; Zhu, Wang & Wang, 2024). Many performance-enhancing techniques, however, lead to increased model complexity. Our research specifically addresses the challenge of improving the accuracy of an existing, proven lightweight architecture (LPRNet) by optimizing its training process with GSS, aiming to achieve a better accuracy-efficiency balance without modifying the inference pipeline.

We selected LPRNet as the base framework for this study due to several key advantages. Firstly, it achieves excellent performance on benchmark LPR datasets while being remarkably lightweight, requiring only approximately 0.34 GFLOPs for a single forward pass (Zherzdev & Gruzdev, 2018). This efficiency makes it highly suitable for deployment on embedded devices with limited computational resources, striking a crucial balance between high recognition accuracy and low resource consumption. Secondly, LPRNet performs end-to-end recognition without requiring prior character segmentation, a common bottleneck in traditional LPR systems. It is characterized by its high accuracy, real-time performance, and inherent support for variable-length license plates. Furthermore, LPRNet has demonstrated its capability for end-to-end training on license plates from different countries, accommodating significant variations in character appearance (Zherzdev & Gruzdev, 2018). As one of the pioneering real-time lightweight LPR algorithms that does not rely on RNNs, its purely convolutional architecture allows it to run efficiently on diverse hardware platforms.

LPRNet has been widely adopted and validated within the LPR research community. Numerous subsequent studies have employed LPRNet either as a baseline for comparison or as a foundational component for further improvements specifically within the domain of license plate recognition. Its application has been predominantly focused on LPR tasks rather than general object recognition, owing to its specialized architecture optimized for recognizing linear sequences of characters typical of license plates. This established track record and its specific design for LPR make it an ideal choice for our work aiming to enhance training methodologies for efficient and accurate license plate recognition.

The backbone network of LPRNet, depicted in Fig. 1, processes an original RGB image to extract discriminative features for license plate recognition. This network is primarily a CNN architecture.

As shown on the left side of Fig. 1, the input image sequentially passes through several key modules:

• CBR blocks: These consist of a 2D Convolutional layer (Conv2d), followed by Batch Normalization (BatchNorm2d) and a rectified linear unit (ReLU) activation function, as detailed on the upper right of Fig. 1.

• MaxPool3d layers: These perform 3D max pooling to reduce spatial dimensions.

• Small basic blocks: These are custom residual-like blocks, detailed in the middle right of Fig. 1. Each block comprises a sequence of Conv2d layers, interspersed with ReLU activations, and concluding with BatchNorm2d and another ReLU. This structure helps in learning more complex features.

• DropOut layers: These are used for regularization to prevent overfitting.

A distinctive feature of LPRNet is its replacement of traditional RNNs for sequence modeling. Instead, LPRNet utilizes a specialized 1 × 13 convolutional kernel (not explicitly shown as a single block in this high-level diagram but is a core part of the Conv2d layers within the architecture, particularly in later stages or the classifier) to capture contextual information across the width of the feature map, effectively acting as a context-bound operator.

Furthermore, LPRNet incorporates global context embedding. As illustrated in the lower right part of Fig. 1, features from different stages or parallel paths (after passing through Average Pool layers) are processed by scaler modules. The scaler module, detailed on the right, normalizes the features using a power function (Power(2)), mean calculation, and division (div), which can be interpreted as a form of local response normalization or feature scaling. These scaled features are then concatenated (Concat) before being fed into the final classifier. This concatenation of globally pooled and scaled features enhances the network’s expressive power by providing a richer set of information to the classifier.

Finally, similar to many sequence recognition models, LPRNet is trained end-to-end using CTC Loss. This loss function is necessary because the length of the network’s encoded output (feature sequence from the backbone) is typically not equal to the variable length of the license plate character sequences. In the field of license plate recognition, the LPRNet model stands out with its relatively high accuracy and lightweight design, and has become one of the most widely used models in the field. Its efficient computational performance and low resource consumption have led to its successful deployment in many embedded devices. In view of this, LPRNet is chosen as the base model in this study, aiming to further improve its recognition accuracy through in-depth optimization of its training process.

Graphically supervised signals

Effectively utilizing supervised signals is a key research area, especially with the rise of big data. Self-supervised Learning has emerged as a prominent approach in recent years. In machine vision, SSL often focuses on extracting supervisory signals directly from input data. A notable example is the MoCo algorithm by He et al. (2022). This method uses contrastive learning and a dynamic dictionary for unsupervised representation learning. Chen et al. (2020) proposed the SimCLR algorithm, which improves the effect of contrast learning by data augmentation with the construction of larger batches. Chen & He (2021) further elaborated the self-supervised algorithm, which solves the problem of gradient collapse by using gradient stopping. He et al. (2022) again re-explored the supervised signals in the image by first randomly masking the image and then predicting the masked portion from the unmasked portion. In the field of natural language processing (Devlin et al., 2018; Radford et al., 2018), although not using the idea of self-supervised learning, it is similar to self-supervised learning, i.e., the model is first trained by a large-scale pre-training task, which allows the model to acquire a certain amount of learning ability in advance by pre-training on an unsupervised, large-scale textual dataset, before fine-tuning it on a specific task.

The license plate recognition task has some special characteristics. Firstly, the resolution of license plate images is small, and small resolution license plate images cannot adapt to most of the auxiliary tasks, which makes self-supervised learning difficult to realize in license plate images; secondly, the signal-to-noise ratio of license plate images in complex scenes is lower compared to that of conventional images, and conventional auxiliary tasks may make the model learn more noise, which in turn reduces the performance of the model.

The research in this article is inspired by self-supervised learning, hoping to mine some additional valuable information in the existing dataset and use this additional information to assist the training of the model, in order to improve the model’s performance at the time of testing. However, the research in this article is not exactly the same as self-supervised learning, and the difference between the research in this article and self-supervised learning is specifically manifested in the following two aspects:

1. The supervisory information of self-supervised learning comes from data, while the supervisory information of this study comes from labels.

2. Self-supervised learning mostly uses the training paradigm of unsupervised pre-training-supervised fine-tuning, while this study merges the pre-training process with fine-tuning, and performs pre-training and fine-tuning at the same time in the form of dual training branches to ensure the effectiveness of the model.

In our pursuit of enhanced performance in license plate recognition (LPR), this study explores the mining of novel supervisory signals from existing information. Conventional deep learning-based LPR, exemplified by models like LPRNet, typically relies on standard supervised learning. In such setups, red green blue (RGB) images serve as input, producing fixed-length feature sequences that are then matched with variable-length text sequence labels through a CTC Loss. The network parameters are subsequently updated via backpropagation, driven entirely by these text-based, semantic-level supervisory signals. While effective, this semantic supervision primarily guides the model towards understanding the textual content.

To further assist the model in learning robust semantic features and thereby improve both recognition accuracy and overall system resilience, we introduce an auxiliary learning mechanism. This mechanism is driven by what we term “graphically supervised signals” (GSS). The core idea is to extract and utilize the inherent graphical information contained within the textual labels themselves to augment the learning process.

For LPR and similar text recognition tasks, text labels inherently possess visual characteristics, primarily defined by their font. By leveraging font rendering, we can transform these textual signals into specific graphical representations—effectively constructing an image from the text. This “image label” then serves as the GSS. These GSS provide a direct, pixel-level target for an auxiliary decoder branch, compelling the shared feature extractor within LPRNet to learn the fine-grained visual characteristics and spatial layout of the characters. This use of label-derived graphical images as explicit supervisory targets for an auxiliary task represents our specific strategy to improve feature learning in the LPR context, complementing the primary semantic supervision provided by the CTC Loss. The concept of GSS and its distinction from purely semantic signals are illustrated in Fig. 2.

The image label of license plate is constructed precisely through fonts, and graphical supervisory signals are introduced in an attempt to obtain supervisory signals in fonts with respect to the graphical features of the text. As shown in Fig. 3, in constructing the image label, we spatially aligns the real training image with the image label. The goal is to make the location of the license plate pixels in the image label as close as possible to the location of the license plate pixels in the original map.

The input to the spatial alignment algorithm is the average image of the training set, and the output is the alignment boundary. The training set is first traversed to find the average image of all license plates, using the average map as an abstract representation of the entire training set. Subsequently, the average map is processed to remove irrelevant information from the image by grayscaling and binarization. Next, the image is projected, respectively, on the horizontal axis and the vertical axis of the two directions. The two projection curves, respectively, set the threshold values of the two directions, and two threshold straight lines can be obtained. Through the threshold straight line and the intersection of the projection curve, you can determine the spatial alignment of the boundary. Then determine the alignment boundary. You can generate the same size with the boundary of the characters, which will be placed in the image in accordance with the alignment boundary.

Auxiliary training

Upsampling decoder

How to effectively utilize graphically supervised signals is a central issue in the task flow. This article draws on the ideas in autoencoder (Michelucci, 2022). Autoencoder belong to unsupervised learning, which predicts the input data through the output data so that the output is as similar as possible to the input, and the purpose of autoencoder is generally to compress the input data by using the hidden layer. Autoencoder put the supervised signals they carry to good use by predicting the input. For the problem of how to utilize the graphical supervisory signal, this article refers to the idea of a autoencoder. The difference is that the autoencoder predicts the input, while this article predicts the image label.

In order to effectively utilize the image label as a supervised signal for training, we design an upsampling decoder in the feature output layer of the original neural network. The construction of the upsampling decoder is accomplished using only a simple three-layer transposed convolution and a projected convolution. The input to the upsampling decoder is the feature map output from the LPRNet backbone, which is used as the input to both the text sequence classifier and the upsampling decoder. The location of the upsampling decoder and its structure are shown in Fig. 4. After processing by the upsampling decoder, the feature map is sampled to the same size as the image label. Subsequently, the output of the upsampling decoder is made as close as possible to the image label by MSE Loss.

MSE Loss in upsampling decoder can be expressed as:

(1) $${\mathrm{L}}_{\mathrm{m}\mathrm{s}\mathrm{e}}={\displaystyle \frac{1}{\mathrm{n}\times \mathrm{h}\times \mathrm{w}}\sum _{\mathrm{i}=1}^{\mathrm{h}}\sum _{\mathrm{j}=1}^{\mathrm{w}}{\left({\mathrm{y}}_{\mathrm{i},\mathrm{j}}-{\hat{\mathrm{y}}}_{\mathrm{i},\mathrm{j}}\right)}^2}$$where, $\mathrm{n}$ denotes the number of license plate samples, $\mathrm{h}$ and $\mathrm{w}$ respectively denotes the height and width of the image label, $\mathrm{y}$ denote the outputs of upsampling decoder, $\mathrm{y}$ denote image label. The closer the upsampling decoder output is to the image label, the smaller the loss.

Algorithmic process

The current conventional license plate recognition process is to input the RGB image of the license plate, get the output of the category sequence through the deep learning model, and then match the category sequence with the label sequence through CTC Loss and calculate the loss.

Suppose the input sequence of CTC is $\mathrm{X}=\left({\mathrm{x}}_1,{\mathrm{x}}_2,\dots ,{\mathrm{x}}_{\mathrm{T}}\right)$, where $\mathrm{T}$ is the CTC input length. the input of CTC is the output of the license plate recognition model, which is a fixed length. Suppose the output sequence of CTC is $\mathrm{Y}=\left({\mathrm{y}}_1,{\mathrm{y}}_2,\dots ,{\mathrm{y}}_{\mathrm{L}}\right)$, where $\mathrm{L}$ is the CTC output length and $\mathrm{L}\le \mathrm{T}$. The purpose of CTC is to find a mapping from $\mathrm{X}$ to $\mathrm{Y}$. CTC generates alignment paths by 1. inserting a blank character $\mathrm{\varnothing }$, at each position in the sequence of output characters to indicate that no character is output at a certain time step. 2. allowing the neighboring output characters to remain the same to indicate continuous output of characters. With these operations, CTC can align fixed-length input sequences with shorter and variable-length output sequences. The output of CTC is actually a probability distribution of a series of labeled sequences, each of which can generate different alignment paths by inserting blank characters and repeating characters. The final output is the probability sum of all possible paths.

The goal of CTC Loss is to maximize the total probability of an alignment path for a sequence Y of correct characters. Let $\mathrm{B}$ denote the function that maps alignment paths to output sequences and $\mathrm{P}(\mathrm{Z}|\mathrm{X})$ denote the probability of generating alignment path $\mathrm{Z}$ given input $\mathrm{X}$, The CTC Loss can be expressed as:

(2) $$\mathrm{P}\left(\mathrm{Y}|\mathrm{X}\right)=\sum _{\mathrm{Z}\in {\mathrm{B}}^{-1}\left(\mathrm{Y}\right)}\mathrm{P}\left(\mathrm{Z}|\mathrm{X}\right)$$where, ${\mathrm{B}}^{-1}\left(\mathrm{Y}\right)$ denotes the set of all alignment paths that can be mapped to the output sequence $\mathrm{Y}$. To compute this probability, CTC computes the probability of each aligned path and sums it by means of dynamic programming.

After calculating the probabilities of all possible paths, the final form of the CTC Loss is the negative log-likelihood function:

(3) $$\mathrm{C}\mathrm{T}\mathrm{C}\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}=-\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{P}\left(\mathrm{Y}|\mathrm{X}\right).$$

The advantage of CTC Loss is that sequence alignment can be performed directly in the training phase without additional alignment labels. It can effectively deal with sequence alignment problems and can automatically learn the alignment corresponding to the target sequence during the training process. In license plate recognition, CTC Loss can be used to train neural network models to improve the accuracy of license plate recognition by optimizing CTC Loss. By introducing CTC Loss, the license plate recognition model can learn the position of the characters and the alignment relationship between the characters, thus improving recognition accuracy.

The characteristic in the algorithm flow in this article is that a new auxiliary training branch is designed, and the auxiliary training branch is used to receive graphic supervised signals. The training process of the model consists of the auxiliary training branch and the regular training branch, and the training process is shown in Fig. 5. The models in the regular training branch and the auxiliary training branch are slightly different, and this article calls the model in the regular training branch the regular model and the model in the auxiliary training branch the auxiliary model. The inputs of the regular model and the auxiliary model are exactly the same, and the backbone part of the model is weight-sharing, which means that the same backbone completes the two training tasks of the two training branches at the same time, i.e., the two training branches are training the same backbone. The difference between the regular model and the auxiliary model is that the regular model inputs the output of the backbone directly into the classifier to get the category sequence, while the auxiliary model inputs the input out of the backbone into the upsampling decoder, which decodes the feature map output from the backbone into an image of the same size as the image label to receive the graphical supervisory signals.

During model training, the two training branches alternate. That is, for a certain batch of data in the training set, the model weights are first updated through the regular training branch, and then using the same data, the model weights are updated through the auxiliary training branch. When both branches have finished updating the weights, the next batch of data is obtained, and the above process is repeated. In the testing phase, only the regular training branch is retained, and the auxiliary training branch is no longer needed. Therefore, no additional computation is added to the model in the testing phase.

To demonstrate that the auxiliary graphical supervision signal effectively contributes to the learning of the primary license plate recognition task, we analyze how gradients from both loss functions influence the parameters of the shared backbone network.

Let ${\theta }_B$ denote the parameters of the shared backbone network $f_B\left(x;{\theta }_B\right)$, which takes an input image x and extracts features. These features are then utilized by two separate heads:

• (1)

  A classifier head $f_C$ (with parameters ${\theta }_C$) for the primary license plate recognition task, optimized using the CTC loss, $L_{ctc}\left({\theta }_B,{\theta }_C\right)$.

• (2)

  An auxiliary decoder head $f_D$ (with parameters ${\theta }_D$) for the graphical reconstruction task, optimized using the mean squared error (MSE) loss, $L_{mse}\left({\theta }_B,{\theta }_D\right)$.

Critically, the backbone parameters ${\theta }_B$ are shared between these two tasks. This means that ${\theta }_B$ appears in the computation graph of both $L_{ctc}$ and $L_{mse}$. Consequently, during backpropagation, gradients from both loss functions will be computed with respect to these shared parameters ${\theta }_B$.

Our proposed training methodology, detailed in Algorithm 1, employs a sequential update strategy for the shared backbone parameters within each training iteration (batch). This process can be described as follows:

Algorithm 1:

Graphically supervised assisted training.

Input: Dataset $D=\left\{\left(x,y_{ctc},y_{\mathrm{m}\mathrm{s}\mathrm{e}}\right)\right\}$, Learning rate $\alpha$

Parameters: Backbone ${\theta }_B$, Classifier ${\theta }_C$, Decoder ${\theta }_D$

Model: $z=f_B\left(x;{\theta }_B\right),p=f_C\left(z;{\theta }_C\right),\hat{y}=f_D\left(z;{\theta }_D\right)$

Losses: $L_{CTC}\left(p,y_{ctc}\right),L_{MSE}\left(\hat{y},y_{mse}\right)$

Output: Optimized Inference Parameters ${\theta }_B^{\ast },{\theta }_C^{\ast }$

1: Initialize ${\theta }_B,{\theta }_C,{\theta }_D$

2: for each epoch do

3: for each batch $B={\left(x_i,y_{ctc,i},y_{mse,i}\right)}_{i=1}^N\subset D$ do

4: // 1. Primary Task Update (CTC)

5: $Z\leftarrow f_B\left(X_B;{\theta }_B\right)$

6: $P\leftarrow f_C\left(Z;{\theta }_C\right)$

7: $L_1\leftarrow Average\left(L_{CTC}\left(P,Y_{ctc,B}\right)\right)$

8: $\left({\theta }_B,{\theta }_C\right)\leftarrow \left({\theta }_B,{\theta }_C\right)-\alpha \ast {\mathrm{\nabla }}_{\left({\theta }_B,{\theta }_C\right)}{L}_1$

9: // 2. Auxiliary Task Update (MSE)

10: $Z^{\mathrm{\prime }}\leftarrow f_B\left(X_B;{\theta }_B\right)$ // Features using updated ${\theta }_B$

11: $\hat{\mathrm{Y}}\leftarrow f_D\left(Z^{\mathrm{\prime }};{\theta }_D\right)$

12: $L_2\leftarrow Average\left(L_{MSE}\left(\hat{\mathrm{Y}},Y_{mse,B}\right)\right)$

13: $\left({\theta }_B,{\theta }_D\right)\leftarrow \left({\theta }_B,{\theta }_D\right)-\alpha \ast {\mathrm{\nabla }}_{\left({\theta }_B,{\theta }_D\right)}{L}_2$

14: end for

15: end for

16: Return ${\theta }_B$, ${\theta }_C$

DOI: 10.7717/peerj-cs.2989/table-101

Let ${\theta }_B^{\left(k\right)}$ be the state of the backbone parameters at the beginning of a training step $k$ for a given batch.

• 1.

  Update from primary task (CTC loss):

• The backbone parameters are first updated based on the gradient from the CTC loss: (4) $$\begin{array}{c}{\theta }_B^{\left(k+1/2\right)}={\theta }_B^{\left(k\right)}-\alpha \ast {\mathrm{\nabla }}_{{\theta }_B}{L}_{ctc}\left({\theta }_B^{\left(k\right)},{\theta }_C\right)\end{array}$$

• where α is the learning rate, and ${\mathrm{\nabla }}_{{\theta }_B}{L}_{ctc}\left({\theta }_B^{\left(k\right)},{\theta }_C\right)$ is the gradient of $L_{ctc}$ with respect to ${\theta }_B$ evaluated at ${\theta }_B^{\left(k\right)}$.

• 2.

  Update from auxiliary task (MSE loss):

• Subsequently, the newly updated backbone parameters ${\theta }_B^{\left(k+1/2\right)}$ are further refined based on the gradient from the MSE loss. The features for this step are computed using $f_B\left(x;{\theta }_B^{\left(k+1/2\right)}\right)$. (5) $$\begin{array}{c}{\theta }_B^{\left(k+1\right)}={\theta }_B^{\left(k+1/2\right)}-\alpha \ast {\mathrm{\nabla }}_{{\theta }_B^{\mathrm{\prime }}}{L}_{mse}\left({\theta }_B^{\left(k+1/2\right)},{\theta }_D\right)\end{array}$$

• where ${\mathrm{\nabla }}_{{\theta }_B^{\mathrm{\prime }}}{L}_{mse}\left({\theta }_B^{\left(k+1/2\right)},{\theta }_D\right)$ denotes the gradient of $L_{mse}$ with respect to the backbone parameters, evaluated at their current state ${\theta }_B^{\left(k+1/2\right)}$.

By substituting Eq. (4) into Eq. (5), we can express the net change to the backbone parameters after one full cycle of updates within the batch:

$${\theta }_B^{\left(k+1\right)}=\left({\theta }_B^{\left(k\right)}-\alpha \ast {\mathrm{\nabla }}_{{\theta }_B}{L}_{ctc}\left({\theta }_B^{\left(k\right)},{\theta }_C\right)\right)-\alpha \ast {\mathrm{\nabla }}_{{\theta }_B^{\mathrm{\prime }}}{L}_{mse}\left({\theta }_B^{\left(k+1/2\right)},{\theta }_D\right)$$

(6) $$\begin{array}{c}{\theta }_B^{\left(k+1\right)}={\theta }_B^{\left(k\right)}-\alpha \ast \left({\mathrm{\nabla }}_{{\theta }_B}{L}_{ctc}\left({\theta }_B^{\left(k\right)},{\theta }_C\right)+{\mathrm{\nabla }}_{{\theta }_B^{\mathrm{\prime }}}{L}_{mse}\left({\theta }_B^{\left(k+1/2\right)},{\theta }_D\right)\right).\end{array}$$

Equation (6) clearly demonstrates that the total update applied to the shared backbone parameters ${\theta }_B$ in a single training step is effectively a combination of the gradients derived from both the primary task ( $L_{ctc}$) and the auxiliary task ( $L_{mse}$). Although the MSE gradient is computed based on slightly updated parameters ( ${\theta }_B^{\left(k+1/2\right)}$), the crucial point is that both supervisory signals contribute distinct gradient information that shapes ${\theta }_B.$

Therefore, the graphical supervision signal, through its $L_{mse}$ gradient, actively participates in and influences the learning process of the shared feature extractor ( $f_B$). This co-supervision encourages the backbone network to learn richer and more robust feature representations that are beneficial not only for reconstructing graphical elements but also for improving the accuracy of the primary license plate recognition task. The optimized backbone parameters ${\theta }_B^{\ast }$ used during inference are thus a result of this synergistic training process.

The practical implementation of this dual-branch training, detailing how these updates are orchestrated, is formally described in Algorithm 1.

Training dataset and evaluation dataset

In this article, we use the CCPD dataset for training, which is a large, diverse, and carefully labeled open-source dataset of Chinese urban license plates (Xu et al., 2018). Data is available at https://github.com/detectRecog/CCPD. After opening the link, find CCPD (ECCV), which can be downloaded directly through the Google Drive link. In order to conform to the license plate recognition problem studied in this article, the dataset is processed in this article. The target region of the license plate is cropped by the corner points of the license plate labeled by the CCPD dataset and corrected by an affine transformation.

Part of the license plate data is shown in Fig. 6. The CCPD dataset provides 100,000 images as a training set and 99,996 images as a validation set for the conventional scenario, and part of the images of the training set and validation set are shown in Fig. 6A. In order to evaluate the recognition ability of the model in various extreme situations, the experiments use four important test sets of extreme situations in CCPD, which are Challenge, Blur, DB, and FN. Challenge (50,003 images) is used to test the performance of the model in the case of challenging license plate images; Blur (20,611 images) is used to test the performance of the model in the case of blurred license plate images; DB (10,132 images) is used to test the model’s performance in the case of too low or too high illumination; and FN (20,967 images) is used to test the model’s performance in the case of low resolution. Some of the images of the four test sets are shown in Fig. 6B. In addition, the CCPD dataset also provides a comprehensive test set, which is a comprehensive sampling of several extreme cases, and the images in this part are corrected by affine transformation.

In addition, in order to evaluate the spatial adaptation ability of the model, the experiments were evaluated using the Rotate and Tilt test sets in CCPD without affine transformation. As shown in Fig. 6C, rotate and Tilt test sets in CCPD with the test set without affine transformation. As shown in Fig. 6C, Rotate (10,053 images) is used to test the performance of the license plate in the case of rotation; the license plate in Tilt (30,216 images) is used to test the performance of the license plate in the case of extreme angles; and finally, the experiment is evaluated on the test set without affine transformation correction, which is used to evaluate the performance of the model without the use of corrective transformations.

Training parameter settings

The experiments were conducted using the processed CCPD training set with the training parameters shown in Table 1.

Assessment of indicators

The evaluation of the license plate recognition task should not only refer to whether the model is correct in recognizing characters, but also whether the model is correct in recognizing the number of characters. Therefore, $\mathrm{T}\mathrm{N}1$ is defined as the number of samples in which the number of characters is recognized incorrectly, $\mathrm{T}\mathrm{N}2$ is defined as the number of samples in which the characters are recognized incorrectly, and $\mathrm{T}\mathrm{P}$ is defined as the number of samples in which the number of characters is recognized correctly and the characters are recognized correctly. The evaluation indexes are introduced: Length Consistency Rate (LCR) and Accuracy. The calculation formula is shown in Eqs. (7) and (8):

(7) $$\mathrm{L}\mathrm{C}\mathrm{R}={\displaystyle \frac{\mathrm{T}\mathrm{N}2+\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{N}1+\mathrm{T}\mathrm{N}2+\mathrm{T}\mathrm{P}}}$$

(8) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{N}1+\mathrm{T}\mathrm{N}2+\mathrm{T}\mathrm{P}}.}$$

The indicator LCR can reflect the accuracy of the number of characters recognized; the closer to 1 means that the number of characters recognized accurately, the greater the number of characters; the indicator Accuracy can reflect the number of characters and the accuracy of character recognition at the same time.

Experimental results and visualization

The LPRNet method based on graphically supervised signal-assisted training proposed in this article is used to do side-by-side comparison experiments with models such as AlexNet (Krizhevsky, Sutskever & Hinton, 2017), ResNet (He et al., 2016), LPRNet, CRNN, EfficientNet (Tan & Le, 2019) and the experimental parameters are shown in Table 1. The accuracy of each model is evaluated on the validation set with different test sets, and the results are shown in Table 2. The code related to the experiments in this article is publicly available and can be accessed via the following GitHub link: https://github.com/GENERjia/License-Plate-Recognition-Based-on-Graphically-Supervised-Signal-Assisted-Training. Or download it via the DOI link: https://doi.org/10.5281/zenodo.15647076.

In order to compare the performance of different models, the computational volume, number of parameters, FPS, and average accuracy of different models are counted, as shown in Table 3.

From Table 3, it can be seen that the proposed algorithm achieves the highest accuracy with the lowest number of parameters and computations. It proves the effectiveness of the proposed algorithm. The computational amount, number of parameters, and average accuracy of each model in Table 3 are visualized as shown in Fig. 7.

The comprehensive comparison of proposed algorithm with other common license plate recognition algorithms is shown in Fig. 8. Figure 8A shows the comparison of the computational volume and accuracy between this article and several common models; the x-axis indicates the computational volume of the model; the size of the diameter of the dots indicates the number of model parameters; and the y-axis indicates the average accuracy in the test set. Figure 8B shows the comparison of the accuracy of several common models on different test sets.

The experiments were recorded for the metrics during the training of CRNN, LPRNet, ResNet, and AlexNet, as shown in Fig. 9.

The results show that LPRNet is much higher than other models in both accuracy and convergence speed.

The experiment uses LPRNet as a baseline and introduces graphically supervised signals on the basis of LPRNet. The experiment records the metrics during the training process of the LPRNet model and proposed algorithm, as shown in Fig. 10.

The results show that the accuracy of proposed algorithm is slightly higher than that of the original model on the validation set, and higher accuracies are obtained on all test sets, which proves that proposed algorithm can show good recognition results. In terms of the convergence speed and loss of the model, proposed algorithm is similar to the original model, i.e., this article’s algorithm does not need more iterations for optimization while improving the accuracy.

In order to more intuitively reflect the improved effect of the algorithm, different license plates from the experimental findings will be shown in the experiment. The results are shown in Fig. 11.

For low light, blur, low resolution, angular offset, motion offset, etc., proposed algorithm can get more and more accurate detection results.

In order to further analyze the results, the model output can be visualized, as shown in Fig. 12A. It is the classification output of LPRNet, and proposed algorithm is more accurate for the identification of blank characters and can distinguish between characters and the interval between them; Fig. 12B is the prediction output of proposed algorithm for the image label. The prediction output of the image label can basically realize the extraction of effective information from the image, and the prediction of the image label can basically react to the classification output situation.

In order to study the recognition of each character in the dataset, we conducted ablation experiments to better validate the effectiveness of proposed algorithm. And because of the uneven balance of the province character categories in the dataset, the experiments were only conducted on the recognition of numeric characters and alphabetic characters, as shown in Table 4.

Visualize the increase of precision, recall, and F1 score indicators for each character using the proposed algorithm in the form of a bar chart, and analyze the increase using a histogram, as shown in Fig. 13. For the recognition of numeric characters, proposed algorithm not only enhances the precision rate of character recognition but also improves the recall rate, enabling it to recognize the characters that the original algorithm can not. As for the alphabetic characters, while proposed algorithm reduces the precision rate of the recognition of some characters, it can greatly improve the recall rate of character recognition so that the F1 score can be improved.

For LPRNet and proposed algorithm, experiments were performed to visualize Grad-CAM (Selvaraju et al., 2017) on both models as shown in Fig. 14 in order to observe the regions of interest on the pictures for each category.

By observing the visualization results, it can be found that proposed algorithm can pay better attention to the correct position of each character without focusing too much on the other regions, which is especially obvious in the model’s understanding of the “-” character, and proposed algorithm can better capture the segmentation and positioning relationship between characters.

In order to evaluate the spatial adaptation ability of the model, the experiments are conducted to evaluate LPRNet and proposed algorithm in the Rotate, Tilt test set, and the test set without affine transformation respectively, and the results are shown in Table 5.

The results show that in the case of large angle change, both LPRNet and proposed algorithm can not achieve better results, and the effect of proposed algorithm is slightly lower than that of LPRNet. While in the case of no affine transformation correction, i.e., test set, proposed algorithm is slightly higher than that of LPRNet, which indicates that this article’s algorithm still achieves better results in the case of small angle change.