Label dependency modeling in Multi-Label Naïve Bayes through input space expansion

Multi-label learning and evaluation metrics

In the realm of multi-label learning, the complexity of the problem is significantly heightened when compared to traditional single-label learning scenarios. The foundational concept of multi-label learning involves instances that are inherently associated with multiple labels simultaneously, rather than a single label. This section will elucidate the formal definition of the multi-label learning problem and delineate the evaluation metrics employed to assess the performance of multi-label classifiers. Let dataset D = {x1, x2, …, xk}, where each xi (for i = 1,2, …, k) is characterized by a set of predictive attributes, labels L = {l 1, l 2, …, l m} where each l, j (for j = 1, 2, …, m), represents a distinct label from the response space. Multi-label classifier 𝔥 is defined as 𝔥: f → 2  l. Each instance in example is related to more than one label from label space. Evaluation metrics used to evaluate multi-label learning are more complicated than traditional single-label learning. Commonly used evaluation metrics are described below:

Binary relevance approach basics

The process of learning from multi-label data involves breaking the multi-label problem down into a number of single-label problems, then learning using a number of base classifiers. Binary relevance (BR) is the most commonly used baseline transformation method because of its simplicity and effectiveness. It has many advantages: (i) simple and independent and (ii) low computational complexity. The binary relevance method uses one–versus–rest strategy and initially transforms the source dataset D into L datasets D_li for each l_i ∈L labels and is defined in Eq. (10). (10)${\displaystyle D_j^{{}^{\prime }}=\left(x_i,\left\{\begin{array}{c}\hfill {\mathit{l}}_i,if{\mathit{l}}_{i}L\hfill \\ \hfill {\mathit{l}}_i,otherwise\hfill \end{array}\right\}\right)}$ Each transformed dataset D_li consists of all examples from source datasets with relevant (positive) or irrelevant (negative) labels associated with single-label l_i, while labels other than l_i are excluded, any supervised binary learning method can be used as the base algorithm. The BR method learns —L— individual {HBR1 ($\left(D_1^{{}^{\prime }}\right)$, HBR2 $\left(D_2^{{}^{\prime }}\right)$, …, HBR—L—($\left(D_{\left|L\right|}^{{}^{\prime }}\right)$} binary classifiers (C li) over the transformed dataset D_li for each label in label space. Individual binary supervised classifier works as in Eq. (11). (11)${\displaystyle H_{BRj}\left(D_j^{{}^{\prime }}\right):\left\{C_{\mathit{l}i}\left\{\left(X\right)\to {\mathit{l}}_i|\neg {\mathit{l}}_i\right\}\forall {\mathit{l}}_i\in L;{\mathit{l}}_i\in \left\{0,1\right\}\right\}}$ The L classifiers are completely independent, implying that the prediction of one classifier does not influence other to work. The predictions from each binary classifier are aggregated to compose the final multi-label prediction. For an unseen instance, label prediction relies on aggregate information supplied by all of the L binary classifiers as in Eq. (12). (12)${\displaystyle Y=\left\{y_j|\prod _{1\le j\le \left|L\right|}{H}_{B{R}_j}\left(D_j^{{}^{\prime }}\right)>0\right\}}$

Stage 1: Feature space reconstruction with meta data

Feature space is extended with existing features along with —L— −1 labels from label space for all labels, i.e., meta data. Meta data used for input space expansion uses all but one response variable from response space. The enhanced feature space carries original feature information and meta data. The attribute space is expanded as in Eq. (15). (15)${\displaystyle D_i^{{}^{\prime }}=\left(\forall x_i\in D_j\right)\cup {\lambda }_j\mid j{\lambda }_j=L-\left\{{\mathit{l}}_j\right\},where{\mathit{l}}_j\in L;j=1,2,3,\dots ,mandF=D_i^{{}^{\prime }}}$ F is the newly expanded feature space with all labels except the label of concern for that current iteration. The augmentation passes label information as meta data in to feature space, allowing the Naïve Bayes to include label dependencies and overcomes the label independence problem by MLNB. In the training phase, the meta data carries actual label values, but in the testing phase the meta data is the predicted value of response variables. However, the size of newly generated feature space may grow exponentially with huge labels which leads to high dimensional input space and eventually induces over fitting. To overcome this over fitting, relevant labels are only permitted to append to the feature space. Correlation among —L— −1 labels is calculated to extract relevant labels. Correlation matrix captures the real relationships among different labels in label space. This matrix provides knowledge to find the labels with richer information. Let existing predictor space is X and —L— −1 labels set is l’ ={(l1 ∪l2, ∪…, ∪ lk-1, ∪lk+1, ∪…, ∪lk)}, then the correlation coefficient matrix of target variables is calculated as in Eqs. (16) and (17). (16)${\displaystyle Corr\left({\mathit{l}}^{\prime }\right)=\frac{cov\left({\mathit{l}}_i^{{}^{\prime }},{\mathit{l}}_j^{{}^{\prime }}\right)}{\sqrt{cov\left({\mathit{l}}_i^{{}^{\prime }},{\mathit{l}}_j^{{}^{\prime }}\right)cov\left({\mathit{l}}_j^{{}^{\prime }},{\mathit{l}}_i^{{}^{\prime }}\right)}},\forall i,j\varepsilon \left\{1,2,3,\dots .,m\right\}}$ (17)${\displaystyle cov\left({\mathit{l}}_i^{{}^{\prime }},{\mathit{l}}_j^{{}^{\prime }}\right)=E\left[\left({\mathit{l}}_i^{{}^{\prime }}-{\mu }_{\mathrm{i}}\right)\left({\mathit{l}}_j^{{}^{\prime }}-{\mu }_{\mathrm{j}}\right)\right]where{\mu }_i=E\left[{\mathit{l}}_j^{{}^{\prime }}\right]}$ E[${\mathit{l}}_{\mathrm{i}}^{\prime }$] is the expected value of ${\mathit{l}}_{\mathrm{i}}^{\prime }$. The responses with correlation p > 0.5 are taken as the response with richer information, others (p < 0.5) are excluded as it may cause noise in the feature space. The targets with richer information are appended to the feature space. This enhanced feature space contains the information of predictors as well as the targets. This feature set includes all the relevant labels in the training phase. This expanded space is the union of multiple continuous and discrete binary predictors. The multi-label Naïve Bayes has to be enhanced to handle this expanded space with heterogeneous predictors.

Stage 2: MLNB parameter modelling

The newly created feature space has a mixture of predictors. One is regular predictors, which have continuous Gaussian or normal distributions; another is discrete, the appended responses that are of binary nature. For each discrete predictor F =f_k, given jth response variable, the probability density function is P (F_i = f_i|L = l_j; Θ_ij). We assume the discrete predictors follow Bernoulli distribution.

For each continuous predictor F =f_i, the probability density function is $g\left(f_i\right|l_j;{\mu }_{ij},{\sigma }_{ij}^2$), where μ_ij and ${\sigma }_{ij}^2$ are the mean and standard deviation of F =f_i. This work assumes the continuous predictors follow Gaussian distribution. The conditional probability density function for new feature space with length of m is defined as follows in Eqs. (18) and (19). (18)${\displaystyle Lengthm=0ton\left(continuouspredictors\right)+\left(n+1\right)toq\left(discretepredictors\right)}$ (19)${\displaystyle Letq=\left(n+1\right)toq}$ By applying Bayesian rule under the assumption of class conditional independence among features and employing the law of total probability, Eq. (20) defines Naïve Bayes’ classifier. (20)${\displaystyle P\left(l_j=1|F\right)=argma{x}_{l_j\varepsilon L;f_i\varepsilon F}\frac{P\left(L=l_j\right)\times \prod _{i=0}^{n}g\left(f_i|l_j;{\mu }_i,{\sigma }_i^2\right)\times \prod _{k=n+1}^{k}P\left(F_k=f_k|L=l_j;{\Theta }_{kj}\right)}{P\left(F=f_i\right)}}$ F is the newly created predictor space with response variables. The above Eq. (20) is the mixed joint density to find the cumulative distribution of two types of random variable where one is continuous random variable and other is discrete random variable. As the denominator P(F = f_i) doesn’t change the result it can be ignored. Pseudo code for training phase is shown in Algorithm 1 .

The probability of response variable given the new attribute space is as follows in Eqs. (21) and (22). (21)${\displaystyle P\left(l_j=1|f_1,f_2,f_3,\dots ,f_n,f_{n+1},f_{n+2},\dots ,f_m\right)=P\left(L=l_j\right)\times \prod _{i=0}^{n}g\left(f_i|l_j;{\mu }_i,{\sigma }_i^2\right)\times \prod _{k=n+1}^{m}P\left(F_k=f_k|L=l_j;{\Theta }_{kj}\right)}$ (22)${\displaystyle P\left(F_i=f_i|L=l_j;{\Theta }_{ij}\right)}$ is the Bernoulli Probability distribution for discrete predictors and the parameter Θ_ij is defined as follows in Eq. (23). (23)${\displaystyle {\Theta }_{ij}=\frac{1+f_{kj}}{q+\sum _{k=n+1}^m{f}_{kj}}}$ $g\left(f_i|l_j;{\mu }_i,{\sigma }_i^2\right)$, is the Gaussian probability distribution for continuous predictors as in Eq. (24) and the parameters μ_i, ${\sigma }_i^2$ are as follows in Eqs. (25) and (26), (24)${\displaystyle P\left(F_i=f_i|L=l_j\right)=g\left(f_i|l_j;{\mu }_{ij},{\sigma }_{ij}^2\right)=\frac{1}{{\sigma }_{ij}\sqrt{2\pi }}{e}^{\frac{-{\left(f_i-{\mu }_{ij}\right)}^2}{2{\sigma }_{ij}^2}}}$ where ${\sigma }_{ij}^2$ and μ_ij are as follows (25)${\displaystyle {\mu }_{ij}=\frac{\sum _{i=0;l_j=1}^n{f}_{ij}}{\left|L=l_j=1\right|}}$ (26)${\displaystyle {\sigma }_{ij}^2=\frac{\sum _{i=0;l_j=1}^n\left(f_{ij}-{\mu }_{ij}\right)}{\left|L=l_j=1\right|}}$ Predicted value of labels in the training phase, used for the input space expansion, are used for the prediction of test instances that have no label values and the pseudo code for testing phase is shown in Algorithm 2 .

Innovative approach

The study presents a new method for multi-label classification known as improved multi-label Naïve Bayes. It allows for the investigation of label correlations by enlarging the input space with meta information from the label space.

Addressing label dependencies

The suggested approach is to record and depict label correlations inside the learning framework. Its goal is to improve the precision of multi-label classification models in this way.

Performance evaluation

The paper evaluates the upgraded iMLNB model’s performance through a thorough empirical examination that uses six benchmark datasets. The performance of the suggested method is contrasted with the conventional multi-label Naïve Bayes algorithm through the use of a variety of assessment criteria. The empirical results indicate that the proposed method outperforms the conventional MLNB algorithm across the evaluated metrics.

Table 10 shows running times of all classifiers used in this research. The training time for iMLNB is O (—Φ— D_li∈L+—L—), Φ = —L— –{l_j}. PCT shows low training time, MLNB follows the above. Proposed method iMLNB exhibits a slightly higher running time than the MLNB method as the feature spce for this method increases as the number of labels increases in label space. However, iMLNB consumes less time as similar to PCT. Figure 3 shows a comparison of running times among existing methods and the proposed methods.

Table 11 consolidates the average performance metrics, providing a holistic view of the effectiveness of the proposed iMLNB system in comparison to the existing MLNB method and other contemporary state-of-the-art approaches. The data underscore the enhanced capability of iMLNB, which consistently outperforms the other methods across various measures. The test time for iMLNB is O (D_li∈L—L—), as the number of labels for the feature space expansion is unknown for the test data. Figure 4 illustrates the competence of the proposed iMLNB method over conventional methods. The iMLNB method outperforms other techniques in classification accuracy, albeit with a slightly higher training time than MLNB. But in average the PCT takes less time to model the training data. Training time for ML-C4.5 is longer than neural network.