Active learning with label quality control

Pool-based active learning

Uncertainty: Let the classification probability vector predicted by model M for sample $x$ be $P=(p_1,\dots ,p_q{)}^T$. Then, the uncertainty method labels samples with high entropy $H(P)$.

(1) $$H(P)=-\sum _{i=1}^q{p}_i\log p_i.$$

When the number of categories $q$ is 2, the uncertainty method degenerates to select the sample with $p_1$ closest to 0.5, which is equivalent to the active learning metric used in the dataset alignment active learning (Ben et al., 2022) method.

CEAL: The cost effective active learning (CEAL) (Wang et al., 2016) method improves the uncertainty method. It not only labels samples with high entropy values but also pseudolabels samples whose entropy values are less than a given threshold. CEAL methods benefit from the use of unlabeled samples, but choosing an appropriate threshold is difficult.

Variation ratio: The variation ratio method trains $c$ neural networks with different random initializations. They have the same network structure and use the same training set. The variance ratio method labels samples with large differences $V(x)$ between the predictions of $c$ neural networks.

(2) $$V(x)=1-\frac{f_m(x)}{c},$$where $f_m(x)$ is the number of predictions falling into the modal class for sample $x$. The variation ratio method performs well but requires additional computing resources to train multiple neural networks.

Core-set: The core-set method selects the labeled samples such that the spheres centered on the samples in the labeled dataset cover all samples. Let the labeled sample set be L and the unlabeled sample set be U. T is the sample set that needs to be labeled, and $\lambda (x_1,x_2)$ is the distance between samples $x_1$ and $x_2$. The core-set method chooses T to minimize

(3) $$\underset{x_i\in U,x_j\in L\cup T}{max}\lambda (x_i,x_j).$$

The core-set method has an intuitive geometric interpretation, but it is often difficult to choose a suitable definition of the distance $\lambda (x_i,x_j)$.

BatchBALD: The BatchBDAL (Kirsch, Van Amersfoort & Gal, 2019) method selects a set of samples to minimize the uncertainty of the model parameters. With appropriate transformations, it chooses to label a set of samples $x_{1:b}$ to maximize

(4) $$\mathcal{H}(y_{1:b}|x_{1:b},D_{train})-E_{p(w|D_{train})}\mathcal{H}(y_{1:b}|x_{1:b},w,D_{train}),$$where $\mathcal{H}$ denotes the conditional entropy. The BatchBDAL method is the original BDAL (Gal, Islam & Ghahramani, 2017) method when $b$ is 1. The BatchBDAL method reduces the probability of labeling multiple similar samples in the same batch, which results in better performance.

Learning loss: The learning loss (Yoo & Kweon, 2019) method adds a loss prediction model to the task model. The model takes the features from the task model as input and then predicts the loss value of the task model for each sample. The learning loss method is widely applicable to various tasks, but its loss prediction model is closely related to the structure of the task model, so the loss prediction model needs to be redesigned when using different task models, which increases the workload of using it.

VAAL: The Variational Adversarial Active Learning (VAAL) (Sinha, Ebrahimi & Darrell, 2019) method first trains an auto-encoder using labeled and unlabeled samples, and then trains a discriminator to predict whether a sample is already labeled based on the features extracted by the auto-encoder. If an unlabeled sample is predicted to be labeled by the discriminator, the variational adversarial method believes that the sample can be well represented by the existing labeled samples and therefore does not deserve to be labeled. The variational adversarial method chooses to label those unlabeled samples that the discriminator is confident are unlabeled.

EADA: The Energy-based Active Domain Adaptation (EADA) (Xie et al., 2022) method is concerned with the application of active learning to knowledge transfer in the target domain. It combines domain features and instance uncertainty to select samples to be labeled. The EADA method also reduces the domain gap by compactly aligning the free energy of the target domain around the source domain through regularization terms. The EADA method reduces the labeling cost required to solve the domain adaptation problem and also expands the application scenario of active learning.

More active learning methods can be found in these surveys (Kumar & Gupta, 2020; Ren et al., 2021).

Active learning with noisy labels

BDAL: The BDAL method calls the active learning process of selecting samples for labeling the forward process, and adds the backward process of selecting samples for inspection. In the backward process, the BDAL method checks whether the label of sample $x_{BL}^{\ast }$ is correct, where

(5) $$x_{BL}^{\ast }=\underset{x\in L}{\arg min}\sum _{i}P(y_i|x;{\theta }_{L\setminus (x,y_i^{\ast })})\cdot \sum _{x^u\in U-x}H(y^u|x^u;{\theta }_{L\setminus (x,y_i^{\ast })}).$$

${\theta }_{L\setminus (x,Y_i^{\ast })}$ represents the model parameters trained after removing sample $x$ and its label $y_i^{\ast }$ in the labeled sample set L. The BDAL method combines active learning and label quality control but requires multiple recalculations of model parameters after the training set changes, which is difficult to apply to deep neural networks.

ALIL: The ALIL method allows annotators to not label indeterminate samples and has satisfactory query complexity in specific cases. However, its computations grow exponentially as the sample dimension increases, which is difficult to apply to sample types such as images.

Label quality control

Simple method: The simple label quality control method gives each sample to multiple annotators for independent labeling, and then takes the most frequent of the multiple label results as the final label result. Although this method increases the labeling cost several times, it is widely used (Li, 2010) because it is easy to implement and is independent of the chosen machine learning method.

Confident learning: The confident learning (Northcutt, Jiang & Chuang, 2021) method first estimates the joint distribution matrix of the noise label and the true label based on the predicted probability output by the model, and then selects the samples that may be mislabeled based on the distribution matrix. For example, the distribution matrix indicates that a sample of class A is mislabeled as B with probability $p$; then, for the sample labeled as B, the confident learning method considers the first $n\ast p$ samples most likely to belong to class A to be mislabeled, where $n$ is the sample size.

GLAD: The Generative model of Labels, Abilities, and Difficulties (GLAD) (Whitehill et al., 2009) method better aggregates the label results of multiple independent annotators. Different from the traditional method that uses the label supported by the most annotators as the final label, the GLAD method establishes a probability model to analyze the difficulty of labeling the sample and the professional level of the annotator, thereby obtaining more accurate label results.

Active learning with label quality control

Both active learning and label quality control consume labeling resources. If all labeling resources are allocated to active learning, the performance of the model may be limited by samples with low label quality. If each sample is labeled multiple times to fully inspect its label quality, the same labeling resources will yield fewer high-quality labeled samples, which will also limit the performance of the model. When the number of labeled samples is small, we tend to increase the number of labeled samples; when the number of labeled samples is sufficient, we tend to improve the label quality of the labeled samples. Therefore, allocating labeling resources to active learning and label quality control in a fixed proportion will not achieve the best performance, and we need to allocate labeling resources adaptively with the training progress.

Active learning and label quality control typically use different metrics for sample selection. Therefore, it is difficult to evaluate whether the benefit of labeling a new sample is greater than the benefit of inspecting an existing label. If active learning and label quality control use the same sample selection metric, we can adaptively allocate labeling resources based on this metric. A human annotator can be considered as a model whose predictions may be wrong due to oversight or other reasons. The goal of label quality control is to inspect samples that the model (human annotator) may predict incorrectly, which is similar to the goal of active learning. Based on this, we propose a sample selection metric that can be used for both active learning and label quality control.

If we let the label space be $\mathcal{Y}$, within this space, the actual label of sample $x$ is $y$, the label given by the annotator is $\overline{y}$, and the prediction given by model M is $\hat{y}$. Let $D(y_1,y_2)$ be the distance in the label space. The smaller the distance is, the closer $y_1$ is to $y_2$. We call $d_{pre}=D(y,\hat{y})$ the predicted distance, $d_{ann}=D(y,\overline{y})$ is the annotation distance, and $d_{obs}=D(\overline{y},\hat{y})$ is called the observation distance. For active learning, we choose to label samples with a large predicted distance $d_{pre}$; to control the label quality, we choose to inspect samples with a large annotation distance $d_{ann}$. Therefore, to combine active learning and label quality control, we choose to label samples with a large $max(d_{pre},d_{ann})$.

Since the actual label $y$ of the sample $x$ is usually unknown, it is difficult to obtain the value of $max(d_{pre},d_{ann})$. If the distance $D(y_1,y_2)$ in the sample space $\mathcal{Y}$ satisfies the triangle inequality, as shown in Fig. 1, then

(6) $$max(d_{pre},d_{ann})\ge \frac{1}{2}{d}_{obs}.$$

Therefore, we choose to label or inspect samples with a large observation distance $d_{obs}$, which represents the lower bound of the predicted distance $d_{pre}$ and the annotation distance $d_{ann}$.

For various common machine learning tasks, such as classification and regression, the label space $\mathcal{Y}$ usually belongs to a high-dimensional vector space. The $L_1$ distance and $L_2$ distance, etc. in the high-dimensional vector space satisfy the triangular inequality, and we usually choose the $L_1$ distance for the stability of the gradient in the training process. For example, in the q-classification problem, the elements in the label space $\mathcal{Y}=\{(p_1,\dots ,p_q{)}^T:0\le p_i\le 1,1\le i\le q;{\sum }_{i=1}^q{p}_i=1\}$ are all possible classification probability vectors. Since the sample is difficult to distinguish, the actual label $y$ is not necessarily a one-hot vector, but the label $\overline{y}$ given by the annotator is often a one-hot vector. We use the $L_1$ distance as the distance metric D in the label space $\mathcal{Y}$, and then define the observation distance $d_{obs}$, annotation distance $d_{ann}$, and prediction distance $d_{pre}$.

Practical algorithm

To apply our proposed method to practical problems, two issues need to be addressed. First, each sample must have a label. Second, neural networks often overfit the training set and thus underestimate the observation distance.

To make each sample have a label, we introduce the pseudolabel method. After updating the labeled sample set, we first train the model on the labeled sample set, and then use the trained model to assign a pseudolabel to each sample in the unlabeled sample set. Pseudolabel methods are widely used in semisupervised learning and are widely applicable to various problems.

To obtain a more accurate observation distance, the model M used to calculate the observation distance of the unlabeled sample $x$ should be trained on a dataset that does not contain $x$. To reduce the number of calculations, we randomly divide the entire dataset (including the labeled dataset and the unlabeled dataset with pseudolabels) into $k$ groups. Each time, training is performed on $k-1$ of the groups, and the trained model is used to calculate the observation distance on the remaining group.

After obtaining the observation distances of all samples, we select the sample with the largest observation distances. If the sample has already been labeled, we re-label it and then update the label based on the results of all previous labels. For example, we update the label to the most frequent label among all previous results. If the sample has not been labeled, we label it and add it to the labeled dataset. The complete process is shown in Algorithm 1, and the flowchart of the proposed method is shown in Fig. 2.

Algorithmic property

Next, we prove that our proposed algorithm is less affected by redundant samples in the dataset.

If the number of folds $k$ is 5, then for any redundant sample group $U_R$, the probability that there are redundant samples from $U_R$ in the sample set to be labeled by Algorithm 1 does not exceed $1/5$. It can be seen that our method avoids labeling redundant samples. Although the proof is only for redundant samples, the experiment in Section “Experiments of Active Learning” shows that our proposed method also performs well for similar samples.

Experiments under noisy label

We first describe the experimental settings used and then compare it to the no quality control, full quality control, and fixed ratio quality control methods.

Compare with no quality control

We first compare with active learning methods without label quality control. These active learning methods directly use low-quality labeled datasets thus obtain more labeled samples. The experimental results in Fig. 3 and Table 1 show that our proposed ALQC method outperforms the other methods. When consuming the same label cost, the ALQC method achieves higher test set accuracy than other methods, thus achieving optimal efficiency in the use of labeling resources. The excellent performance of the ALQC method shows that ignoring label quality control and directly using low-quality labels does not achieve the best performance. This highlights the necessity of combining active learning with label quality control. The CEAL method is limited by the large number of false pseudolabels and therefore performs poorly, which also illustrates the importance of label quality control in active learning.

Table 1:

Test set accuracy of active learning methods without label quality control on the MNIST dataset.

Label cost	ALQC	CEAL	Core	Rand	Var	Unc
100	0.7967	0.7862	0.7877	0.7868	0.7877	0.7833
200	0.9107	0.8468	0.8565	0.8712	0.8867	0.8472
300	0.9447	0.8747	0.9073	0.9050	0.9328	0.9021
400	0.9623	0.8710	0.9320	0.9209	0.9561	0.9212
500	0.9732	0.8600	0.9400	0.9315	0.9669	0.9403
600	0.9777	0.8526	0.9497	0.9353	0.9742	0.9539
700	0.9819	0.8433	0.9535	0.9420	0.9773	0.9609
800	0.9849	0.8393	0.9567	0.9456	0.9801	0.9660
900	0.9865	0.8316	0.9635	0.9470	0.9817	0.9713
1,000	0.9876	0.8276	0.9652	0.9510	0.9831	0.9759

DOI: 10.7717/peerj-cs.1480/table-1

Compare with fully quality control

We then compare our proposed method with active learning methods that use a simple label quality control method for label quality control. The simple label quality control method requests multiple labels for each sample until a category appears twice in the label results. Then we select that category as the final label result. If the method does not stop after five times of labeling, one of the five labeling results is randomly selected as the final labeling result. All samples are first run through this simple method to control the labeling quality and then used for active learning, so the average number of annotations required to obtain a labeled sample is 2.1. The accuracy of the labeling results obtained from this process is greater than 99.9%. The results in Fig. 4 and Table 2 show that our proposed method also outperforms other methods. The ALQC method uses fewer labeling resources than other methods to achieve the same test set accuracy, thus reducing labeling costs. The excellent performance of the ALQC method shows that forcing all samples to be labeled with high quality does not achieve the best performance either. Therefore, it is necessary to properly allocate labeling resources between active learning and label quality control.

Table 2:

Test set accuracy of active learning methods with the simple label quality control method on the MNIST dataset.

Label cost	ALQC	CEAL	Core	Rand	Var	Unc
100	0.7967	0.7076	0.7090	0.7073	0.7135	0.7071
200	0.9107	0.7947	0.8043	0.8325	0.8335	0.8177
300	0.9447	0.8061	0.8595	0.8806	0.9012	0.8621
400	0.9623	0.7683	0.8921	0.9017	0.9213	0.8826
500	0.9732	0.7310	0.9045	0.9145	0.9445	0.9110
600	0.9777	0.7211	0.9237	0.9261	0.9575	0.9231
700	0.9819	0.7173	0.9331	0.9345	0.9649	0.9385
800	0.9849	0.7100	0.9447	0.9397	0.9690	0.9479
900	0.9865	0.7031	0.9482	0.9425	0.9728	0.9520
1,000	0.9876	0.6985	0.9531	0.9476	0.9754	0.9607

DOI: 10.7717/peerj-cs.1480/table-2

Compare with fixed ratio quality control

Finally, we compare the proposed method with methods that allocate labeling resources to active learning and label quality control in a fixed ratio. In each iteration, we allocate 10%, 20%, and 50% of the labeling resources to the confident learning method to improve label quality, and the remaining labeling resources to the uncertain method to select samples worth labeling. The experimental results in Fig. 5 and Table 3 show that our proposed method still outperforms the other methods. This shows that simply allocating a fixed ratio of labeling resources to improve labeling quality does not achieve the best performance, and highlights the importance of adaptively allocating labeling resources. Our proposed ALQC method is based on the same sample selection metric, and thus can adaptively allocate labeling resources to achieve the best performance. As can be seen in Fig. 6, the ALQC method allocates almost 0% of the labeling resources to inspect the quality of existing labels at the beginning, and adaptively increases the allocation percentage to about 15% as the labeling sample size increases, demonstrating a good adaptive labeling resource allocation capability.

Table 3:

Test set accuracy of the uncertainty method combined with the confident learning method on the MNIST dataset.

Label cost	ALQC	Unc	10% CL + 90% Unc	20% CL + 80% Unc	50% CL + 50% Unc
100	0.7967	0.7833	0.8008	0.7916	0.8030
200	0.9107	0.8472	0.8667	0.8569	0.8700
300	0.9447	0.9021	0.9100	0.9099	0.9049
400	0.9623	0.9212	0.9258	0.9282	0.9314
500	0.9732	0.9403	0.9431	0.9432	0.9427
600	0.9777	0.9539	0.9535	0.9507	0.9476
700	0.9819	0.9609	0.9638	0.9619	0.9552
800	0.9849	0.9660	0.9681	0.9684	0.9606
900	0.9865	0.9713	0.9734	0.9716	0.9665
1,000	0.9876	0.9759	0.9767	0.9732	0.9692

DOI: 10.7717/peerj-cs.1480/table-3

The above experiments show that adaptive allocation of labeling resources can achieve better performance compared to no quality control, full quality control, and fixed ratio quality control. By allocating the labeling resources to the highest value samples to be labeled and samples to be inspected, our proposed ALQC method improves the utilization of labeling resources and thus achieves the best performance.

Experiments of active learning

Assuming that the labeling is absolutely correct, we compare the proposed method with various active learning methods on the MNIST dataset and the CIFAR-10 dataset to observe its active learning performance, and experimentally observe the redundancy resistance of our proposed method.

First, we experiment with the MNIST dataset. The experimental settings are the same as in the “Experimental Settings” section. Instead of simulating the noisy noise, we use the labels provided by the dataset directly. We add comparisons with the recent active learning methods: “LLAL” is the learning loss method, “VVAL” is the variational adversarial active learning method, and “BBALD” is the BatchBALD method. We use the code provided by the authors of the relevant articles for our experiments and try to make the training settings as similar as possible. The network structure, the data augmentation methods, the way and number of initial sample sets are selected, the number of labeled samples per iteration, and the forbidding of additional labeled validation sets are all consistent with our experimental settings. The experimental results are shown in Fig. 7 and Table 4. It can be seen that our proposed ALQC method outperforms other methods on the active learning task without noise labeling.

We also compare the performance of the ALQC method using the $L_1$ distance metric and the $L_2$ distance metric. The experimental results in Fig. 8 show that the ALQC method is robust to the choice of distance metric. The ALQC method relies only on the support of the distance metric for the triangular inequality and does not require a specially designed distance metric.

Then we conduct the experiments on the CIFAR-10 dataset. Most of the experimental settings are the same as in Section “Experimental Settings”, with the following main differences. We switch to use the ResNet-18 (He et al., 2016) network. We also use the SGD optimizer and update the batch size to 128, then train 200 epochs. The learning rate is adjusted to 0.2 times the previous one after 60, 120, and 160 epochs. We use RandomCrop, RandomHorizontalFlip, and Cutout (DeVries & Taylor, 2017) methods for data augmentation. We randomly select 1,000 samples from the training set in the CIFAR-10 dataset as the initial set of labeled samples, and for each iteration, we label or inspect 1,000 samples until we have consumed 10,000 labeling resources. The experimental results are shown in Fig. 9 and Table 5. It can be seen that our proposed method also outperforms the other methods.

Although our proposed method uses pseudolabels to calculate the observation distance, the performance improvement does not come from the use of pseudolabels. In the above experiments, our proposed method outperforms the CEAL method that also uses pseudolabels. This is because our proposed method keeps improving the labeling quality of the training set, but the CEAL method accumulates more and more incorrect pseudolabels, which limits its performance. We also add pseudolabels to the random and uncertain methods and compare them with our method. The experimental results are shown in Fig. 10. It can be seen that our proposed method still outperforms the other methods when using pseudolabels. Thus, the performance improvement mainly stems from the good redundancy resistance rather than simply using pseudolabels.

Since our proposed method has excellent redundancy resistance, it reduces the labeling resources wasted on labeling multiple redundant samples and thus achieves better active learning performance. To further verify the redundancy resistance of our proposed method, we simulate the presence of redundant samples in the dataset. We randomly select 20% of the samples in the training set of the MNIST dataset to replicate, add random noise from a normal distribution N (0, 0.01) to the replicated samples, and then add them to the original dataset. Training starts with 100 randomly selected labeled samples and labels 100 new samples at each iteration until a total of 1,000 labeled samples are obtained. We test the redundant sample rate of the selected sample set under different numbers of folds $k$ and compare it with its 95% confidence interval. As shown in Fig. 11, with the increase of the number of folds, the redundancy rate of the selected sample set becomes lower. This is consistent with the conclusion in Theorem 1 that our method has good anti-redundancy performance.

The above experiments show that our proposed method has excellent active learning performance due to good redundancy resistance, and thus can achieve excellent performance in active learning tasks under noise labeling.

Experiments of label quality control

We randomly select 1,000 samples from the training set in the MNIST dataset and label them with 95% accuracy. We then use our proposed method to inspect samples that may be mislabeled to improve the quality of the labels. We inspect 100 existing labels per iteration until 1,000 inspections are performed. We compare it with the simple label quality method and the confidence learning method mentioned in the “Related Work” section. The result is shown in Fig. 12. It can be seen that our proposed method can select samples that may be mislabeled more effectively than the simple method and achieve similar performance as the confidence learning method.

The experimental results in this section show that our proposed method can allocate labeling resources to the most valuable tasks and has good active learning capability with labeling quality capability, thus achieving leading performance in active learning tasks under noisy labeling.

Problem description

AOI is an optical-based detection system that uses images captured by a camera to detect defects in printed circuit boards (PCB). The AOI system is constantly receiving so many new unlabeled samples that there are not enough resources to label all of them. Therefore, it is necessary to use the active learning method to select samples worthy of being labeled.

The redundancy of the AOI dataset is extremely serious. There are multiple similar defect samples in the dataset. Theorem 1 shows that our proposed method can resist the interference of redundant samples, so we apply it to the AOI system.

Method adaptation

Since too many redundant samples waste computational resources, we modify the original method workflow. Samples with a small observation distance mean that the current model can accurately classify them, so they are more likely to be redundant samples. After computing the observation distances, we permanently remove samples with sufficiently small observation distances from the unlabeled dataset, thereby speeding up the method. The adapted workflow is shown in Fig. 13.

Results and analysis

There are a total of 3,100 samples in the AOI dataset, and all experiments start with 100 randomly selected labeled samples. In each iteration, 100 unlabeled samples are labeled, and 100 samples are removed from the unlabeled dataset. The experimental results are shown in Fig. 14 and Table 6. It can be seen that the proposed method outperforms other methods in this real application. In addition, the performance of the ALQC method steadily improves as the label cost increases, and its robust performance is more suitable for the needs of practical industrial applications.