Positive Unlabeled Learning Selected Not At Random (PULSNAR): class proportion estimation without the selected completely at random assumption

Introduction

¹ In a standard binary supervised classification problem, the classifier (e.g., decision trees, support vector machines, etc.) is given training instances $\mathcal{X}$ with features $x$ and their labels $y=0$ (negative) or $y=1$ (positive). The classifier learns a model $f:\mathcal{X}\to 0,1$, which classifies an unlabeled instance as positive or negative based on $x$. It is often challenging, expensive, and even impossible to annotate large datasets in real-world applications (Jaskie, Elkan & Spanias, 2019), and frequently only positive instances are labeled. Unlabeled instances with their features can be classified via positive and unlabeled (PU) learning (Jaskie, Elkan & Spanias, 2019; Elkan & Noto, 2008). Some of the PU learning literature focuses on improving classification metrics, and others focus on the problem of estimating the fraction, $\alpha$, of positives among the unlabeled instances. Although this work focuses on the latter, calibration and enhancing classification performance are also addressed.

PU learning problems abound in many domains (Jaskie & Spanias, 2019). For instance, in electronic healthcare records, the lack of a diagnosis code does not confirm a patient’s negative disease status, as negatives are not routinely recorded nor have billing codes, making traditional supervised learning impractical. Much medical literature is dedicated to estimating disease incidence and prevalence but contends with incomplete medical assessment and recording. The potential to assess disease incidence without costly in-person assessment or chart reviews could have substantial public health benefits. In market research, one typically has a medium-sized set of positives, say of customers or buyers of a product, has a set of attributes over both the positives and a large population of unlabeled people of size N, and wishes to establish the size of the addressable market, $\alpha N$.

The majority of PU learning algorithms use the selected completely at random (SCAR) assumption, which states that the labeled positive examples are randomly selected from the universe of positives. That is, the labeling probability of any positive instance is constant (Elkan & Noto, 2008). This assumption may fail in real-world applications. For example, in email spam detection, positive instances labeled from an earlier time period could differ from later spam due to adaptive adversaries. In patient health data, for a given disease, labeled positive examples might be skewed to patients with severe symptoms who sought medical attention, while many mild cases remain undiagnosed and unlabeled. In fraud detection systems, labeled positive examples (fraudulent transactions) are those confirmed through investigations. Many fraudulent transactions go undetected and unlabeled. In social media bot detection, labeled bots are those reported by users or identified by automated systems. Many sophisticated bots go unnoticed and remain unlabeled.

Although some PU learning algorithms have shown promising performance on different machine learning (ML) benchmark SCAR datasets, the development of PU learning algorithms to estimate the extent of undercoding in large and highly imbalanced selected not at random (SNAR) real-world data remains an active research area. Class imbalance in a PU setting generally means the number of unlabeled instances is large compared to the labeled positive examples. Also, current PU learning approaches have rarely explored how to calculate well-calibrated probabilities for PU examples in SCAR and SNAR settings. In addition, few PU algorithms have been assessed when $\alpha$ is small (≤5%), where performance is expected to suffer.

In this article, we propose a PU learning approach to estimate $\alpha$ when positives are SCAR or SNAR, and evaluate its performance in simulated and real data. We assess the performance with class imbalance in both medium-sized and large datasets and over a rigorous $\alpha$ range. Our contributions are summarized as follows:

• 1)

  We propose PULSCAR, a PU learning algorithm for estimating $\alpha$ when the SCAR assumption holds. It uses kernel density estimates of the positive and unlabeled distributions of ML probabilities to estimate $\alpha$. The algorithm employs the beta distribution to estimate density and introduces an objective function that provides a rapid, robust estimate of $\alpha$.

• 2)

  We propose PULSNAR, a PU learning algorithm for estimating $\alpha$ when the positives are SNAR. It employs a clustering approach to group SNAR positives into subtypes and estimates separate $\alpha$ for each subtype using PULSCAR on positives from each cluster and all unlabeled instances. The final $\alpha$ is calculated by aggregating the $\alpha$ estimated for each subtype.

• 3)

  We propose methods to calibrate the probabilities of PU examples to their true (unknown) labels and improve the classification performance in SCAR and SNAR settings.

Related work

Early PU learning methods (Yu, Han & Chang, 2004; Yu, 2005; Wang et al., 2006) generally followed a two-step heuristic: (i) identify strong negative examples from the unlabeled set, and then (ii) apply an ML algorithm to given positive and identified negative examples. In contrast, Fung et al. (2005) extracted high-quality positive and negative examples from the unlabeled set and then applied classifiers to those data. Some recent work iteratively identifies better negatives (Luo et al., 2021), or combines negative-unlabeled learning with unlabeled-unlabeled learning (Hammoudeh & Lowd, 2020).

Studies predominantly centered around the SCAR assumption focused on estimating the proportion of positives among the unlabeled examples with or without the PU classifier. Elkan & Noto (2008) introduced the SCAR assumption and proposed a PU method to estimate the mixture proportion under the SCAR assumption. By partially matching the class-conditional density of the positive class to the input density under Pearson divergence minimization, Du Plessis & Sugiyama (2014) estimated the mixture coefficient. Jain et al. (2016) proposed a nonparametric class prior estimation technique, AlphaMax, using two-component mixture models. The kernel embedding approaches KM1 and KM2 (Ramaswamy, Scott & Tewari, 2016) showed that the algorithm for mixture proportion estimation converges to the true prior under certain assumptions. Estimating the class prior through decision tree induction (TIcE), Bekker & Davis (2018) provides a lower bound for label frequency under the SCAR assumption. Using the SCAR assumption, DEDPUL (Ivanov, 2020) estimates $\alpha$ by applying a compute-intensive EM-algorithm to probability densities; the method also returns uncalibrated probabilities.

Other studies focused on learning a classifier from PU data. Lee & Liu (2003) converts PU data learning into a noisy learning problem by designating all unlabeled instances as negatives. They employ a linear function to learn from these noisy examples using weighted logistic regression. Confident learning (CL) (Northcutt, Jiang & Chuang, 2021) combines the principle of pruning noisy data, probabilistic thresholds to estimate noise, and sample ranking. Multi-positive and unlabeled learning (Xu et al., 2017) extends PU learning to multi-class labels. Oversampling the minority class (Chawla et al., 2002; Yan et al., 2019) or undersampling the majority class are not well-suited approaches for PU data due to contamination in the unlabeled set; Su, Chen & Xu (2021) uses a re-weighting strategy for imbalanced PU learning.

Recent studies have focused on labeling/selection bias to address the SCAR assumption not holding. Bekker, Robberechts & Davis (2019), Gerych et al. (2022) used propensity scores to address labeling bias and improve classification. Using the propensity score, based on a subset of features, as the labeling probability for positive examples, Bekker, Robberechts & Davis (2019) reduced the Selected At Random (SAR) problem into the SCAR problem to learn a classification model in the PU setting. The “Labeling Bias Estimation” approach was proposed by Gong et al. (2021) to label the data by establishing the relationship among the feature variables, ground-truth labels, and labeling conditions.

Problem formulation and algorithms

In this section, we explain: i) the SCAR and SNAR assumptions, ii) our PULSCAR algorithm for SCAR data and PULSNAR algorithm for SNAR data, iii) kernel density and bandwidth estimation techniques, and iv) a method to find the number of clusters in the labeled positive set. Our method to calibrate probabilities and enhance classification performance using PULSCAR/PULSNAR is in “Algorithm for Calibrating Probabilities” and “Improving Classification Performance with PULSCAR and PULSNAR”, sections in the Appendix, respectively.

Positive and Unlabeled Learning Selected Completely At Random (PULSCAR) algorithm

Given any supervised probabilistic classifier, $\mathcal{A}(x,y)$, let $D_p$, $D_n$, and $D_u$ be probability density functions (PDFs) corresponding to the probability distribution of positives, negatives, and unlabeled respectively. Let $\alpha$ be the unknown proportion of positives in the unlabeled, then

(2)

$$\begin{array}{rl}& D_u=\alpha D_p+(1-\alpha )D_n\\ & \Rightarrow D_u\ge \alpha D_p,\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{c}\mathrm{e}0\le \alpha \le 1\mathrm{a}\mathrm{n}\mathrm{d}{D}_n\mathrm{i}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{n}\text{-}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\\ & \Rightarrow 0\le \alpha D_p\le D_u,\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{c}\mathrm{e}{D}_p\mathrm{i}\mathrm{s}\mathrm{n}\mathrm{o}\mathrm{n}\text{-}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\end{array}$$

From relation (Eq. (2)), a key observation is that $\alpha D_p$ should not exceed $D_u$, ensuring that the $\alpha$ is bounded. However, the true $\alpha$ is usually difficult to identify (Ivanov, 2020; Blanchard, Lee & Scott, 2010). Therefore, PULSCAR aims to estimate the largest $\alpha$ value ( ${\alpha }_{max}$) that ensures $D_{u_i}-\alpha D_{p_i}$ remains non-negative throughout the range, i $\in$ [0 …1]. We calculate $D_u$ and $D_p$ for $n\mathrm{\_}bins$ equidistant points within the range [0, 1]. Thus, $D_u$ and $D_p$ are vectors that represent discrete approximations of the continuous PDFs. The variable $i$ represents each of these $n\mathrm{\_}bins$ points. The ${\alpha }_{max}$ is given by:

$${\alpha }_{max}=min\left(\frac{D_{u_i}}{D_{p_i}}\right)\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{l}\mathrm{l}i,\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}{D}_{p_i}\ne 0$$

Leveraging relation (Eq. (2)), we define the following objective function (Eq. (3)) that PULSCAR utilizes to estimate $\alpha$:

(3)

$$f(\alpha )=log(|min(D_u-\alpha D_p)|+ϵ)\text{,}whereϵ=|min(D_p)|\text{i}fmin(D_p)\ne 0\text{, else }ϵ={10}^{-10}$$

Explanation of function components: Since $D_u$ and $D_p$ are vectors, we identify the smallest difference between $D_u$ and $\alpha D_p$ for each $\alpha$ value in [0 …1] using the min() function. This ensures the objective function is not significantly affected by potential outliers in $D_u$ or $D_p$. The absolute value ensures we consider the magnitude of this minimum difference, regardless of whether $D_u$ is greater or less than $\alpha D_p$. It also prevents taking the logarithm of a negative value. The log() function makes the objective function more sensitive to small differences between $D_u$ and $\alpha D_p$, and the inclusion of $ϵ$ prevents $log(0)$ when $|min(D_u-\alpha D_p)|$ approaches zero.

How PULSCAR estimates $\alpha$: We train a supervised probabilistic classifier (e.g., XGBoost) treating all labeled positives as class 1 and all unlabeled examples as class 0 to estimate the class 1 probabilities for all instances. We then calculate $D_p$ and $D_u$ using the beta kernel density method on estimated class 1 probabilities for positive and unlabeled examples, respectively. The intuition behind the objective function is that $|min(D_u-\alpha D_p)|$ approaches zero at an $\alpha$ point where $\alpha D_p$ equals $D_u$, causing the objective function to tend toward $-\mathrm{\infty }$, see Fig. 1D. Consequently, when $|min(D_u-\alpha D_p)|$ is not zero, there is a maximum change in the value of the objective function (from $-\mathrm{\infty }$ to some value). Thus, this $\alpha$ value creates a sharp change in the objective function, which usually coincides with ${\alpha }_{max}$ because, at ${\alpha }_{max}$, $D_u$ is closest to $\alpha D_p$. To estimate $\alpha$, PULSCAR searches for the point of maximum change in the objective function using its slope. We calculate the value of the objective function, $f({\alpha }_i)$, for each ${\alpha }_i$ in [0 …1] with a step size of 0.0001. We then apply the finite difference method to approximate the slope of the objective function at each $\alpha$ value in this range. The $\alpha$ value corresponding to the maximum absolute difference in slope is selected as the estimated $\alpha$. The objective function may not be convex for some datasets, i.e., there could be multiple points of sharp change in the objective function. In such cases, we select the $\alpha$ corresponding to the first point of sharp change (closest to zero) because the term $D_u-\alpha D_p$, representing the estimated PDF of negative examples, must be non-negative. Selecting a point of sharp change in the objective function beyond the first one may yield a larger $\alpha$ that can make $D_u-\alpha D_p$ negative, violating PDF constraints (Fig. 1B). This approach eliminates the need for implementing an iterative solver technique, accounting in part for the speed of our algorithm. The value of $\alpha$ can also be determined visually by plotting the objective function (Fig. 1D); the sharp change in the plot represents the value of $\alpha$. Algorithm 1 shows the pseudocode of the PULSCAR algorithm to estimate $\alpha$ using the objective function based on probability densities. Algorithm 2 is a subroutine to compute the beta kernel bandwidth. Full source code for our algorithms is available at GitHub (https://github.com/unmtransinfo/PULSNAR/).

Algorithm 1 :

PULSCAR algorithm.

Input: X = $X_p\cup X_u$, s = $s_p\cup s_u$, n_bins

{‘s’ indicates assigned labels (labeled positives: s=1, unlabeled: s=0); n_bins is number of bins}

Output: estimated α

1: p $\leftarrow \mathcal{A}$(X, s) { $\mathcal{A}$(X, s) returns estimated class 1 probabilities for all PU examples}

2: $p_u\leftarrow$ p[s==0] {s==0 indicates unlabeled indices}

3: $p_p\leftarrow$ p[s==1] {s==1 indicates labeled positive indices}

4: estimation_range ← [0, 0.0001, 0.0002,…, 1.0]

5: bw ← estimate_bandwidth_pu(p, n_bins)

6: $D_u\leftarrow$ beta_kernel(pu, bw, n_bins)

7: $D_p\leftarrow$ beta_kernel(pp, bw, n_bins)

8: $\epsilon \leftarrow |\mathrm{m}\mathrm{i}\mathrm{n}(D_p)|$

9: if ε = 0 then

10:    $ϵ\leftarrow {10}^{-10}$

11: end if

12: len ← length(estimation_range)

13: selected_range ← estimation_range[2:len]

14: $\alpha \leftarrow$ estimation_range

15: f(α) $\leftarrow \mathrm{l}\mathrm{o}\mathrm{g}(|\mathrm{m}\mathrm{i}\mathrm{n}(D_u-\alpha D_p)|+ϵ)$

16: d ← f’(α)

17: i ← where the absolute difference in d is the maximum

18: return selected_range[i]

DOI: 10.7717/peerj-cs.2451/table-101

Algorithm 2 :

Estimate bandwidth (estimate_bandwidth_pu).

Input: preds, n_bins

{‘preds’ indicates the classifier-estimated class 1 probabilities}

Output: bandwidth

1: bw ∈ [0.01, 0.5]

2: $D_{hist}\leftarrow$ histogram(preds, n_bins, density=True)

3: $D_{beta}\leftarrow$ beta_kernel(preds, bw, n_bins)

4: return optimize(MeanSquaredError(Dhist, Dbeta))

DOI: 10.7717/peerj-cs.2451/table-102

Our PULSCAR algorithm (Algorithm 1) uses a histogram bin count heuristic to generate a histogram-derived density, then optimizes the beta distribution kernel bandwidth to best fit that density. The approach relies on three key elements: the use of the beta kernel for density estimates, the histogram bin count, and the kernel bandwidth. The beta kernel has important properties, described below, for fitting distributions over the interval $[0\dots 1]$. The histogram bin count parameter is set using standard heuristics for fitting a histogram to data, and defines an array of evenly spaced numbers over the interval $[0\dots 1]$, where the beta kernel density estimates are computed to optimize fitting the histogram. The bandwidth parameter determines the width of the beta kernel, balancing the need for narrower kernels to precisely estimate the distribution with many data points against wider kernels to avoid overfitting when data are sparse. The following three subsections detail these parameters and explain how they are determined.

Beta kernel density estimation

A beta kernel estimator is used to create a smooth density estimate of both the positive and unlabeled ML probabilities, generating distributions over [ $0\dots 1$], free of the problematic boundary biases of kernels (e.g., Gaussian) whose range extends outside that interval, adopting the approach of Chen (1999). Another problem with (faster) Gaussian kernel density implementations is that they often use polynomial approximations that can generate negative values in regions of low support, dramatically distorting $\alpha$ estimates, which require non-negative probability distribution estimates. The beta PDF is as follows (Virtanen et al., 2020):

(4)

$$h(x,a,b)=\frac{\mathrm{\Gamma }(a+b)x^{a-1}{(1-x)}^{b-1}}{\mathrm{\Gamma }(a)\mathrm{\Gamma }(b)},$$for x $\in$ [0,1], where $\mathrm{\Gamma }$ is the gamma function, $a=1+\frac{z}{bw}$ and $b=1+\frac{1-z}{bw}$, with $z$ the bin edge (a value in an array of evenly spaced $n\mathrm{\_}bins$ numbers over the interval [0, 1]), and $bw$ the bandwidth.

Histogram bin count

Our implementation supports five well-known methods to determine the number of histogram bins: square root, Sturges’ rule, Rice’s rule, Scott’s rule, and Freedman–Diaconis (FD) rule (Alxneit, 2020). Let ‘pr’ be the classifier-estimated class 1 probabilities for all PU examples and ‘n’ be the number of PU examples. The formulae to count the number of bins using these methods are as follows:

• i)

  Square root method: $\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{s}=\sqrt{n}$

• ii)

  Sturges’ rule: $\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{s}=1+{\log }_2(n)$

• iii)

  Rice’s rule: $\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{s}=2\times n^{1/3}$

• iv)

  Scott’s rule:

  $\mathrm{h}=3.5\times {\displaystyle \frac{\mathrm{S}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{a}\mathrm{r}\mathrm{d}\mathrm{D}\mathrm{e}\mathrm{v}\mathrm{i}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(pr)}{n^{1/3}}}$, $\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{s}={\displaystyle \frac{\mathrm{m}\mathrm{a}\mathrm{x}(\mathrm{p}\mathrm{r})-\mathrm{m}\mathrm{i}\mathrm{n}(\mathrm{p}\mathrm{r})}{\mathrm{h}}}$

• v)

  Freedman–Diaconis (FD) rule:

  $\mathrm{h}=2\times {\displaystyle \frac{\mathrm{I}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{Q}\mathrm{u}\mathrm{a}\mathrm{r}\mathrm{t}\mathrm{i}\mathrm{l}\mathrm{e}\mathrm{R}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}(\mathrm{p}\mathrm{r})}{n^{1/3}}}$, $\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{s}={\displaystyle \frac{\mathrm{m}\mathrm{a}\mathrm{x}(\mathrm{p}\mathrm{r})-\mathrm{m}\mathrm{i}\mathrm{n}(\mathrm{p}\mathrm{r})}{\mathrm{h}}}$

Beta kernel bandwidth estimation

The bandwidth of the kernel is the smoothing parameter in kernel density estimation that affects the shape of the estimated distribution curve. A broader bandwidth results in a smoother and more generalized curve, whereas a narrower bandwidth produces a more fine-grained curve. Kernel bandwidth selection can also significantly influence $\alpha$ estimates: too narrow of a bandwidth can result in outliers driving poor estimates, and too wide of a bandwidth prevents distinguishing between distributions. We compute a histogram-derived density using the aforementioned bin count heuristic and beta kernel density estimate at those bin centers using the ML probabilities of both the positive and unlabeled examples. We find the minimum of the mean squared error (MSE) between the histogram and beta kernel densities using the scipy differential_evolution() optimizer (Storn & Price, 1997), solving for the best bandwidth in the range [0.01…0.5]. The differential_evolution() algorithm is a stochastic direct search method to find the minimum of a multivariate function without requiring gradient information. The estimated bandwidth is chosen for kernel density estimation in the PULSCAR algorithm. All experiments herein use MSE as the error metric, but alternatively, the Jensen-Shannon distance can be employed.

Positive and Unlabeled Learning Selected Not At Random (PULSNAR) algorithm

We propose a new PU learning algorithm (PULSNAR) to estimate the $\alpha$ in SNAR data, i.e., labeled positives are not selected completely at random. PULSNAR uses a divide-and-conquer strategy for the SNAR data to convert a SNAR problem into several sub-problems. It employs clustering techniques to divide SNAR positives into clusters, where each cluster predominantly contains one subtype of positives. Subsequently, it applies the PULSCAR algorithm to the positives from each cluster along with all unlabeled data to estimate the $\alpha$ for each subtype. The final $\alpha$ is computed by summing the $\alpha$ returned by the PULSCAR algorithm for each subtype of positives.

(5)

$$\alpha ={\alpha }_1+{\alpha }_2+...+{\alpha }_c,c=\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{l}\mathrm{u}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{s}$$

Figure 2 visualizes the PULSNAR algorithm, and Algorithm 3 provides its pseudocode.

Algorithm 3:

PULSNAR algorithm.

Input: X = $X_p\cup X_u$, s = $s_p\cup s_u$, n_bins

{‘s’ indicates assigned labels (labeled positives: s=1, unlabeled: s=0); n_bins is number of bins}

Output: estimated α

1: feature_importance ( $v_1\dots v_k$), imp_features ( $x_1\dots x_k$) $\leftarrow \mathcal{A}$(X, s) { $\mathcal{A}$(X, s) returns important features and their importance (gain scores)}

2: $x_1^{\prime }\dots x_k^{\prime }\leftarrow x_1{v}_1\dots x_k{v}_k$ {scale important features by their gain score}

3: $X_p^{\prime }\leftarrow X_p$[ $x_1^{\prime }\dots x_k^{\prime }$] {select labeled positives with only scaled important features}

4: clusters $C_1...C_c\leftarrow$ GMM( $X_p^{\prime }$) {divide labeled positives into c clusters}

5: $\alpha \leftarrow$ 0

6: for j in $C_1...C_c$ do

7:    $X^{\prime }\leftarrow X_p$[j] $\cup X_u$ {select labeled positives from cluster and all unlabeled}

8:    $s^{\prime }\leftarrow s_p$[j] $\cup s_u$

9:    $\alpha \leftarrow \alpha +\mathrm{P}\mathrm{U}\mathrm{L}\mathrm{S}\mathrm{C}\mathrm{A}\mathrm{R}(X^{\prime },s^{\prime },\mathrm{n}\mathrm{\_}\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{s})$

10: end for

11: return α

DOI: 10.7717/peerj-cs.2451/table-103

Clustering rationale

Suppose both the positive and unlabeled sets contain positives from $c$ subclasses ( $1\dots c$). With selection bias (SNAR), the subclass proportions will vary between the sets, and thus, the PDF of the labeled positives cannot be scaled by a uniform $\alpha$ to estimate positives in the unlabeled set. The smallest subclass would drive an $\alpha$ underestimate with PULSCAR. To address this, we perform clustering on the labeled positives using the key features that differentiate positives from the unlabeled ones. This assumes that these features also correlate with the factors driving selection bias within each subclass. By dividing positives into $c$ clusters based on these features, we establish that within each cluster, the selection bias is uniformly distributed, implying that all members of a cluster are equally likely to appear in the unlabeled set. Consequently, PU data comprising examples from one cluster and the unlabeled set approximates the SCAR assumption. The uniformity allows for the inference of a cluster-specific $\alpha$ using PULSCAR, which can be summed over all clusters to provide an overall estimate of positives in the unlabeled set (Fig. 2).

Determining the number of clusters in the positive set

Determining the “optimal number of clusters” is a challenging problem, as evidenced by the NP-hardness of optimal k-means clustering in the literature (Mahajan, Nimbhorkar & Varadarajan, 2012). Our approach to determining the number of clusters via BIC is one of the widely used heuristics and approximation methods (Zhao, Hautamaki & Fränti, 2008; Ezugwu et al., 2022; Fraley & Raftery, 1998). We build an XGBoost model on all positive and unlabeled examples to determine the important features and their gain scores. A gain score measures the magnitude of the feature’s contribution to the model. We select all labeled positives and then cluster them on those features scaled by their corresponding gain score, using scikit_learn’s Gaussian mixture model (GMM) method. To establish the number of clusters (n_components), we iterate n_components over $1\dots c$ (e.g., $c$ = 25) and compute the Bayesian information criterion (BIC) (Vrieze, 2012) for each clustering model. We use max_iter = 250, and covariance_type = “full”. The other parameters are used with their default values. We implemented the “Knee Point Detection in BIC” algorithm, explained in Zhao, Hautamaki & Fränti (2008), to find the number of clusters in the labeled positives.

Experimental methods

We evaluated our proposed PU learning algorithms in terms of $\alpha$ estimates, probability calibration (“Experiments and Results”), and six classification performance metrics (“Experiments and Results”). We used real-world ML benchmark datasets and synthetic data for our experiments. For real-world data, we used KDD Cup 2004 particle physics (Caruana, Joachims & Backstrom, 2004), Diabetes health indicators (UCI ML Repository, 2024), Magic gamma telescope (Bock, 2007), Electrical grid stability (Arzamasov, 2018), Wilt (Johnson, 2014), and Mice protein expression (Higuera, Gardiner & Cios, 2015) as SCAR datasets and Anuran calls (Colonna et al., 2017), dry bean (UCI Machine Learning Repository, 2020), Room occupancy estimation (Singh & Chaudhari, 2023), Smartphone (Davis & Owusu, 2016), Letter recognition (Slate, 1991), and Statlog (Shuttle) (UCI Machine Learning Repository, 2022) as SNAR datasets. Synthetic (SCAR and SNAR) datasets were generated using the scikit-learn function make_classification() (Pedregosa et al., 2011). We used XGBoost as a binary classifier in our proposed algorithms. We evaluated PU algorithms on imbalanced SCAR and SNAR datasets. In some datasets, we kept the number of unlabeled examples greater than the positive examples while in others, we kept it lesser. By setting the number of unlabeled examples to be less than positive examples, we were able to test the PU algorithms for a higher proportion of positives among unlabeled examples in ML benchmark datasets. To address class imbalance, we used the scale_pos_weight parameter of XGBoost to scale the weight of the labeled positive examples by the factor $s=\frac{|U|}{|P|}$. We also compared our methods with recently published methods for PU learning: KM1 and KM2 (Ramaswamy, Scott & Tewari, 2016), TIcE (Bekker & Davis, 2018) and DEDPUL (Ivanov, 2020). Some recent studies (Bekker, Robberechts & Davis, 2019; Gerych et al., 2022) investigating scenarios where SCAR does not hold do not focus on $\alpha$ estimation or probability calibration. Therefore, we opted to exclude them from our comparison. KM1, KM2, and TIcE algorithms were not scalable on large datasets, so we used relatively smaller synthetic and ML benchmark datasets to compare our methods with these methods. We compared PULSNAR with only DEDPUL on large synthetic datasets (“DEDPUL vs. PULSNAR: Alpha Estimation”). Also, Ivanov (2020) previously demonstrated that DEDPUL outperformed KM and TIcE algorithms on several ML benchmark and synthetic datasets.

Results

Synthetic datasets

Figure 3 shows the $\alpha$ estimated by PU learning algorithms for synthetic datasets. TIcE overestimated $\alpha$ for all fractions in both SCAR and SNAR datasets. For SCAR datasets, only PULSCAR returned close estimates for all fractions; DEDPUL overestimated for 1%; KM1 overestimated for $\ge$20%; KM2 underestimated for 50% and overestimated for 1%. For SNAR datasets, only PULSNAR’s estimates were close to the true $\alpha$; other algorithms either overestimated or underestimated $\alpha$ for some or all fractions. Given the five subclasses in the simulated SNAR data, PULSNAR always estimated either five or six clusters, resulting in accurate overall $\alpha$ estimates when the subclass $\alpha$ estimates were summed. “DEDPUL vs. PULSNAR: Alpha Estimation” shows the $\alpha$ estimated by DEDPUL and PULSNAR on large SNAR datasets with different class imbalances. As the class imbalance increased, DEDPUL underestimated $\alpha$, especially for larger fractions. The estimated $\alpha$ by the PULSNAR method was close to the true fractions across all sample sizes and class imbalances.

ML benchmark datasets

SCAR data

Figure 4 shows the $\alpha$ estimated by PU learning algorithms for SCAR benchmark datasets. The KDD cup and diabetes datasets were too large to execute KM1 and KM2. These methods produced overestimated $\alpha$ values for the magic gamma and electric grid datasets, and for the wilt and mouse protein datasets, $\alpha$ values were either overestimated or underestimated for some fractions. TIcE overestimated $\alpha$ for the KDD cup and mouse protein datasets, while for other datasets, it produced either overestimated or underestimated $\alpha$ values for some fractions. PULSCAR and DEDPUL both provided estimates that were close to the true answers for all fractions and datasets, but DEDPUL overestimated $\alpha$ for 1% for all datasets.

SNAR data

Figure 5 shows the $\alpha$ estimated by PU learning algorithms for SNAR benchmark datasets. For all datasets, only PULSNAR provided close estimates for all fractions, while other algorithms (KM1, KM2, TIcE, DEDPUL, and PULSCAR) either overestimated or underestimated $\alpha$ for some or all fractions. For fractions $\ge$15%, KM1, KM2, TIcE, DEDPUL, and PULSCAR usually underestimated $\alpha$ across all datasets. The $\alpha$ estimates of KM1 were inaccurate for all fractions except 1%, across all datasets. KM2 and TIcE showed inconsistent results, overestimating or underestimating even for smaller fractions ( $<$10%) in some datasets. DEDPUL overestimated alpha for 1% in all datasets. By contrast, our PULSCAR method provided $\alpha$ estimates that were close to the true answers for smaller fractions ( $<$10%).

Discussion and conclusion

This article presented novel PU learning algorithms for estimating the proportion of positives ( $\alpha$) among unlabeled examples in both SCAR and SNAR data with and without class imbalance. By utilizing the law of total probability to define the objective function (Eq. (3)) and a beta kernel to estimate probability densities, the PULSCAR algorithm estimates an upper bound on the fraction of positives in the unlabeled set under the assumption of SCAR and that the classes are separable. Although theoretically an upper bound, in practice, our estimates were all close to the actual ground truth $\alpha$, albeit with greater percent overestimation with smaller $\alpha$ values. We observed in nearly all of our simulations on SCAR data that the $\alpha$ estimates were above ground truth, suggesting the PULSCAR algorithm is trustworthy for bounding $\alpha$ from above. Overestimation may occur due to errors in estimating the continuous density functions with finite sample sizes, and issues of lack of support between positives and unlabeled in the data generation process due to random sampling.

In our experiments, we demonstrated that our PULSCAR method outperformed other state-of-the-art methods, such as KM1, KM2, and TIcE, for estimating $\alpha$ on both synthetic and real-world SCAR datasets (Figs. 3A, 4). The rather poor performance of KM1, KM2, and TIcE in some datasets seems to be linked to the difficulty of the classification problems. Our simulated data used a low class separability (class_sep = 0.3), and experiments we did with higher class separability (class_sep = 0.9) resulted in better $\alpha$ estimation for those algorithms. To highlight the effectiveness of our algorithms under challenging conditions, we focused on scenarios with low class separability (class_sep = 0.3). We ran experiments with higher separability (class_sep = 0.9) in only a few simulated cases before discontinuing that line of inquiry. Consequently, results based on high separability were excluded from this article. The performance of the PULSCAR and DEDPUL methods was comparable on SCAR datasets, given their similar theoretical foundations, as well as the use of powerful classifiers (XGBoost for PULSCAR, CatBoost for DEDPUL). However, we observed that DEDPUL tends to have large overestimates for smaller fractions (e.g., for 1%, DEDPUL estimated an $\alpha$ of $\ge$2% for all datasets). The version of PULSCAR presented in this article uses the beta distribution, outperforming earlier prototypes that used the Gaussian distribution. This may account for PULSCAR’s superiority over DEDPUL for 1%, which uses the Gaussian kernel for density estimation.

Our PULSNAR method demonstrated superior performance over all other methods (KM1, KM2, TIcE, and DEDPUL) for estimating $\alpha$ across both synthetic and real-world SNAR datasets (Figs. 3B, 5, “DEDPUL vs. PULSNAR: Alpha Estimation”). Additionally, the PULSNAR method showed better classification performance compared to DEDPUL when evaluated on SNAR datasets (“DEDPUL vs. PULSNAR: Classification Performance”), particularly for larger fractions ( $\ge$20%). Interestingly, as the number of positive subclasses increased, the SCAR-based methods produced poor $\alpha$ estimates on SNAR data (Figs. 3B, 5), indicating that PU learning methods based on the SCAR assumption are not suitable for SNAR data.

Probability calibration plays a crucial role in several classification tasks, such as medical diagnosis and financial risk assessment. It helps determine the optimal probability threshold for decision-making, which ultimately enhances the reliability and usefulness of ML models (Silva Filho et al., 2023; Pleiss et al., 2017). For instance, in a binary classification scenario where model estimations determine whether a patient has a disease, the selection of an optimal probability threshold for making informed decisions relies on having calibrated probabilities. Our study presented a novel approach to calculating calibrated probabilities for positive and unlabeled examples, which, to the best of our knowledge, has not been explored in existing PU methods. Our experimental results demonstrated that both PULSCAR and PULSNAR methods significantly improved probability calibration for SCAR and SNAR datasets, respectively (“Experiments and Results”).

While not the main focus of this work, we also demonstrated that after applying PULSCAR and PULSNAR, classifier performance improved significantly (“Experiments and Results”) for SCAR and SNAR data, respectively.

An important contribution of this work is the speed of $\alpha$ estimation. Our experimentation showed that the KM1, KM2, and TIcE algorithms exhibited scalability issues. This observation aligns with the findings of Garg et al. (2021), who noted the underperformance of these techniques in high-dimensional scenarios and scalability issues with large datasets. While we evaluated PULSCAR/PULSNAR against these methods using moderately sized datasets, it is plausible that their inherent limitations with data size and high dimension contributed to inaccurate $\alpha$ estimates for some of our test datasets. Considering the scalability limitations and long execution time associated with KM1, KM2, TIcE, and DEDPUL (Table 2), these algorithms would be challenging to use for real-world applications involving millions of records and thousands of features. On the contrary, our preliminary work (Kumar et al., 2022) suggests that PULSCAR and PULSNAR are suitable for processing very large high-dimensional datasets ( $n>1M$, 150,000+ features). While in principle, other SCAR-based algorithms can be substituted for PULSCAR into our PULSNAR clustering approach, due to overestimation/underestimation issues (Figs. 3A, 4), scalability concerns, maintenance issues, and execution time constraints (Table 2), employing KM1, KM2, TIcE, and DEDPUL for estimating $\alpha$ in SNAR settings was not implemented in this work.

A limitation of our approach is that it relies on knowing whether the data are SCAR or SNAR to effectively select between PULSCAR or PULSNAR for $\alpha$ estimation. The issue is the “knee point detection” cluster determination approach relies on three consecutive points in the BIC curve to identify the knee (angle). As the first point cannot have a knee, this approach always returns more than one cluster. We tried other methods, such as minimum BIC, difference in BIC, or other clustering algorithms, but saw performance degradation in PULSNAR $\alpha$ estimates. When more than one clusters are used with PULSNAR on SCAR data, it can result in PULSNAR overestimating $\alpha$ as two near-identical positive types cannot be distinguished and get counted more than once. This $\alpha$ overestimation would adversely affect classification performance. In the future, we aim to enhance the PULSNAR algorithm so that it can be applied without knowing whether the data are SCAR or SNAR. Importantly, preliminary work indicates that PULSNAR $\alpha$ estimation is robust to overestimating the number of clusters in SNAR data: additional clusters beyond the known number of subtypes just slightly increase $\alpha$ estimates. The primary focus of our study was the estimation of $\alpha$ under the SCAR assumption failure. Preliminary findings on cluster overestimation, though informative, were excluded to streamline the narrative and emphasize the core findings.

The utility of $\alpha$ estimation might be illustrated with the problem of estimating the fraction of bots among social media accounts, an active area of ML research (Efthimion, Payne & Proferes, 2018; Heidari, James & Uzuner, 2021). If a trustworthy $\alpha$ upper bound estimate for the bot fraction among unlabeled accounts came to, say, 20%, business decisions could be made based on accounts comprising at least 80% real people. However, one might expect that positively detected bots are not SCAR, and thus, PULSNAR provides a means for estimating $\alpha$ despite that. However, with the PULSNAR clustering heuristic, we cannot guarantee an upper bound as we can in the SCAR case. Nevertheless, our experiments show that PULSNAR far outperformed SCAR-based methods with SNAR positives. In addition, we have found high utility with PULSNAR in identifying uncoded self-harm in electronic health records (Kumar et al., 2022). In that setting, PULSCAR underestimated $\alpha$ due to self-harm positives being SNAR as expected, but with PULSNAR, as confirmed with chart review, we had accurate $\alpha$ estimation as well as good calibration. Notably, that effort identified 14 and 15 self-harm clusters in two independent datasets respectively, demonstrating PULSNAR’s effectiveness in subclass clustering for accurate $\alpha$ estimates. Since the SCAR assumption frequently does not hold in real-world data, robust calibrated model estimations along with $\alpha$ estimates in SNAR settings using PULSNAR opens up new horizons in PU Learning.

Appendix

Algorithm for calibrating probabilities

Under the SCAR assumption, the relationship $p(y=1|x)=\frac{p(s=1|x)}{p(s=1|y=1)}$ holds, where $p(s=1|y=1)$ is a constant (Lemma 1 from Elkan & Noto (2008)). Although this relationship provides a theoretical basis, it does not guarantee that the estimated probabilities, $p(y=1|x)$, are well-calibrated, particularly in imbalanced datasets. To address this limitation, we propose a novel algorithm designed to effectively calibrate probabilities in both balanced and imbalanced datasets.

Algorithm 4 shows the complete pseudocode to calibrate the classifier-estimated class 1 probabilities of PU examples. Once $\alpha$ is known, we seek to transform the original class 1 probabilities so that their sum is equal to $\alpha |U|$ among the unlabeled or $|P|+\alpha |U|$ among positive and unlabeled, and that they are well-calibrated. Our approach is to probabilistically flip the assigned label of $\alpha |U|$ unlabeled examples to positive (from 0 to 1) in such a way as to match the PDF of labeled positives across 100 equispaced bins over $[0\dots 1]$, then fit a logistic or isotonic regression model on those labels versus the classifier-estimated class 1 probabilities to generate the transformed probabilities. To determine the number of unlabeled examples that need to be flipped in each bin, we compute the normalized histogram density, $D\mathrm{\_}hist$, for the labeled positives with 100 bins and then multiply $\alpha |U|$ with $D\mathrm{\_}hist$.

The unlabeled examples are also divided into 100 bins based on their estimated probabilities. Starting from the bin with the highest probability (p = 1), we randomly select m examples and flip their labels from 0 to 1, where m is the number of unlabeled examples that need to be flipped in the bin. If the number of records ( $n_1$) that need to be flipped in a bin is more than the number of records ( $n_2$) present in the bin, the difference ( $n_1-n_2$) is added to the number of records to be flipped in the next bin, resulting in $\alpha |U|$ flips.

After flipping the labels of $\alpha |U|$ unlabeled examples from 0 to 1, we fit an isotonic or sigmoid regression model on the classifier-estimated class 1 probabilities with the updated labels to obtain calibrated probabilities.

The above calibration approach applies to both SCAR and SNAR data. For the SNAR data, the PULSNAR algorithm divides labeled positive examples into c clusters and estimates the $\alpha$ for each cluster. For each cluster, the classifier-estimated class 1 probabilities of the examples (positives from the cluster and all unlabeled examples or only unlabeled examples) are calibrated using the estimated $\alpha$ for the cluster. Since, for each cluster, PULSNAR uses all unlabeled examples, each unlabeled example has c classifier-estimated/calibrated probabilities. The final classifier-estimated/calibrated probability of an unlabeled example is calculated using the following Eq. (6):

(6)

Experiments and results

We used synthetic SCAR and SNAR datasets and the KDD Cup SCAR dataset to test our calibration algorithm.

SCAR datasets: After estimating the $\alpha$ using the PULSCAR algorithm, we applied Algorithm 4 to calibrate the classifier-estimated probabilities. To calculate the calibrated probabilities for both positive and unlabeled (PU) examples, we applied isotonic regression to the classifier-estimated class 1 probabilities of PU examples with labels of positives and updated labels of unlabeled (of which $\alpha |U|$ were flipped per Algorithm 4). We applied isotonic regression to the unlabeled’s estimated probabilities with their updated labels to calculate the calibrated probabilities only for the unlabeled.

SNAR datasets: Using the PULSNAR algorithm, the labeled positive examples were divided into c clusters. For each cluster, after estimating the $\alpha$, Algorithm 4 was used to calibrate the classifier-estimated probabilities. To calculate the calibrated probabilities for positives from a cluster and all unlabeled examples, we applied isotonic regression to their classifier-estimated class 1 probabilities with labels of positives from the cluster and updated labels of unlabeled (of which ${\alpha }_j|U|$ were flipped for cluster $j=1\dots c$, see Algorithm 4). We applied isotonic regression to the unlabeled’s estimated probabilities with their updated labels to calculate the calibrated probabilities only for the unlabeled. Thus, each unlabeled example had c calibrated probabilities. We computed the final calibrated probability for each unlabeled example using Eq. (6).

Figures 6–11 show the calibration curves generated using the unblinded labels and isotonically calibrated (red)/uncalibrated (blue) probabilities. When both positive and unlabeled examples were used to calculate calibrated probabilities, the calibration curve followed the y = x line (well-calibrated probabilities). When only unlabeled examples were used, the calibration curve for 1% did not follow the y = x line, presumably due to the ML model being biased toward negatives, given the small $\alpha$. Also, the calibration curves for the SCAR data followed the y = x line more closely than the calibration curves for the SNAR data. It is due to the fact that the final probability of an unlabeled example in the SNAR data is computed using its c probabilities from c clusters. So, a poor probability estimate from even one cluster can influence the final probability of an unlabeled example. Importantly, the sum of the calibrated probabilities adds up to $|P|+\alpha |U|$ when both positive and unlabeled examples are calibrated, and to $\alpha |U|$ when only unlabeled examples are calibrated, despite some of the calibration curves appearing to be skewed above the target $y=x$.

Improving classification performance with pulscar and pulsnar

Algorithm 5 shows the complete pseudocode to improve classification performance with PULSCAR and PULSNAR. The algorithm returns the following six classification metrics: Accuracy, AUC-ROC, Brier score (BS), F1, Matthew’s correlation coefficient (MCC), and Average precision score (APS). The approach to enhancing the classification performance is as follows:

Using PULSCAR: After estimating the $\alpha$, the classifier-estimated (classifier trained on labeled positive and unlabeled data) class 1 probabilities of only unlabeled examples are calibrated using Algorithm 4. The calibrated probabilities of the unlabeled examples are sorted in descending order, and the labels of top $\alpha |U|$ unlabeled examples with the highest calibrated probabilities are flipped from 0 to 1 (probable positives). We then train and test a second ML classifier (XGBoost) with five-fold CV using the labeled positives and probable positives as class 1, and the remaining unlabeled examples as class 0. The classification performance metrics are calculated using the second classifier-estimated class 1 probabilities and the true labels of the examples.

Using PULSNAR: The PULSNAR algorithm divides the labeled positive examples into c clusters. For each cluster, after estimating ${\alpha }_j$ for $j$ in $1\dots c$, the classifier-estimated (classifier trained on labeled positive and unlabeled data) class 1 probabilities of only unlabeled examples are calibrated using Algorithm 4. Since each unlabeled example has c calibrated probabilities, we compute the final calibrated probability for each unlabeled example using the Eq. (6). The final $\alpha$ is calculated by summing the ${\alpha }_j$ values over the $c$ clusters. The final calibrated probabilities of the unlabeled examples are sorted in descending order, and the labels of top $\alpha |U|$ unlabeled examples with the highest calibrated probabilities are flipped from 0 to 1 (probable positives). We then train and test a second ML classifier (XGBoost) with five-fold CV using the labeled positives and probable positives as class 1, and the remaining unlabeled examples as class 0. The classification performance metrics are calculated using the second classifier-estimated class 1 probabilities and the true labels of the examples.

Experiments and results

We applied Algorithm 5 to synthetic SCAR and SNAR datasets to get the performance metrics for the XGBoost model with PULSCAR and PULSNAR, respectively. The classification performance metrics were also calculated without applying the PULSCAR or PULSNAR algorithm, in order to determine the improvement in the classification performance of the model. The experiment was repeated 40 times by selecting different train and test sets using 40 random seeds to compute the 95% confidence interval (CI) for the metrics.

Figures 12 and 13 show the classification performance of the XGBoost model with/without the PULSCAR or PULSNAR algorithm on synthetic SCAR and SNAR data, respectively. The classification performance using PULSCAR or PULSNAR increased significantly over XGBoost alone. As the proportion of positives among the unlabeled examples increased, the performance of the model without PULSCAR or PULSNAR (blue) worsened significantly more than when using PULSCAR or PULSNAR.

Dedpul vs. pulsnar: alpha estimation

Public implementations of the PU learning methods KM1, KM2, and TIcE were not scalable; they either failed to execute or would have taken weeks to run the multiple iterations required to obtain confidence estimates for large datasets. We thus could not compare our method with KM1, KM2, and TIcE on large datasets and used only DEDPUL for comparison. Importantly, it was previously demonstrated that the DEDPUL method outperformed these three methods on several ML benchmark and synthetic datasets (Ivanov, 2020).

We compared our algorithm with DEDPUL on synthetic SNAR datasets with different fractions (1%, 5%, 10%, 20%, 30%, 40%, and 50%) of positives among unlabeled examples. In our experiments, we observed that class imbalance (ratio of majority class to minority class) could affect the $\alpha$ estimates. So, we used four different sample sizes: 1) positive: 5,000 and unlabeled: 5,000; 2) positive: 5,000 and unlabeled: 25,000; 3) positive: 5,000 and unlabeled: 50,000; 4) positive: 5,000 and unlabeled: 100,000. For each sample size and fraction, we generated 20 datasets using sklearn’s make_classification() method with random seeds 0–19 to compute 95% CI. We used class_sep = 0.3 for each dataset to create difficult classification problems. All datasets were generated with 100 attributes and six labels (0–5), defining ‘0’ as negative and 1–5 as positive subclasses. The positive set contained 1,000 examples from each positive subclass in all datasets. The unlabeled set comprised k% positive examples with labels (1–5) flipped to 0 and (100-k)% negative examples. The unlabeled positives were markedly SNAR, with the five subclasses comprising 1/31, 2/31, 4/31, 8/31, and 16/31 of the unlabeled positives.

Figure 14 shows the $\alpha$ estimates by DEDPUL and PULSNAR on synthetic SNAR data. For smaller true fractions (1%, 5%, 10%), DEDPUL returned close $\alpha$ estimates, but for larger fractions (20%, 30%, 40%, and 50%), it underestimated $\alpha$. Also, as the class imbalance increased, the performance of DEDPUL dropped, especially for larger true fractions. The estimated $\alpha$ by the PULSNAR method was close to the true $\alpha$ for all fractions and sample sizes.

Dedpul vs. pulsnar: classification performance

The main focus of this study was on comparing $\alpha$ estimates provided by different PU methods. Hence, we did not extensively compare the classification performance of our methods with other PU methods. Moreover, methods such as KM1, KM2, and TIcE do not return estimated probabilities, which hinders the comparison of classification performance with these methods. Also, the classification performance of PULSCAR and DEDPUL on SCAR datasets was comparable. Consequently, we compared the classification performance of PULSNAR and DEDPUL using two SNAR datasets: smartphone and dry bean. Future studies would focus on a more comprehensive comparison of classification performance and exploring further enhancements in this regard.

For DEDPUL, leveraging its estimated $\alpha$, we flipped the labels of $\alpha |U|$ unlabeled examples with the highest estimated probabilities from 0 to 1. Subsequently, we employed the XGBoost model with five-fold CV, treating true positives and probable positives as positive (class 1) examples, and probable negatives as negative (class 0) examples to generate estimated class 1 probabilities for all examples in the datasets. The classification performance metrics were then computed using these XGBoost estimated probabilities and the true labels of the examples.

For PULSNAR, we employed the method discussed in “Improving Classification Performance with PULSCAR and PULSNAR” to calculate the classification performance metrics.

Figures 15 and 16 show the classification performance of the XGBoost models with/without identifying probable positives using DEDPUL and PULSNAR. As the true fraction of positives among the unlabeled set increased for both datasets, the $\alpha$ estimates by DEDPUL were not close to true fractions, leading to a decrease in its classification performance. No significant drop in the classification performance of PULSNAR was observed, as the $\alpha$ estimates by PULSNAR were close to true answers.