Detect feature edges for diagnosis of bacterial vaginosis

Vaginal microbiome datasets

The dataset (http://www.ncbi.nlm.nih.gov/bioproject/?term=PRJNA208535) was originally reported by Ravel et al. (2013). Ravel et al. (2013) sequenced vaginal communities collected daily for ten weeks from 25 women diagnosed with symptomatic BV (SBV: n = 15 women), asymptomatic BV (ABV: n = 6), or healthy (HEA: n = 4). In total, Ravel et al. (2013) sequenced 1,657 samples (median = 67 per woman) and obtained 420 8,757,681 high-quality sequenced reads of the V1–V3 hypervariable region of 16S-rRNA genes, with a median of 5,093 reads per sample. These datasets had been generated >10 years ago, they were reanalyzed using the amplicon sequence variant (ASV) method. The softwares Mothur (version 1.39.5), VSEARCH (version 2.3.4) and USEARCH (version 10.0.240) were used for data analysis. First, we used the command “pcr.seqs” in Mothur to remove the primers of the sequence. We then used VSEARCH to remove sequence redundancy and set the parameter minuniquesize of VSEARCH to 1. We used USEARCH to reduce the noise of the clean sequence to generate a representative ASV sequence and set the parameter minisize of USEARCH to the default value of 8. We used VSEARCH again to produce the ASV table and set the matching ratio to 1 (i.e., complete matching). Finally, we used blastn combined with EzBioCloud database to annotate the obtained ASV. Although ASV can be annotated using the Silva database, the 16S sequence of EzBioCloud database has undergone several manual corrections, which is more accurate than the annotation in the Silva database.

Feature selection algorithms

Feature selection aims to find the optimal subset of features. Feature selection can be used to eliminate irrelevant or redundant features, reduce the number of features, filter out features related to class information, and improve model accuracy. The general process of feature selection:

• Generate subsets: search for feature subsets and provide feature subsets for the evaluation function;

• Evaluation function: evaluate the quality of the feature subset;

• Stopping criteria: related to the evaluation function, generally a threshold; the search can be stopped after the evaluation function reaches a certain standard;

• Verification process: verify the validity of the selected feature subset on the verification dataset.

DT (Bramer, 2007) and RF (Robnik-Šikonja & Kononenko, 2003) feature selection algorithms, which belong to the supervised feature selection method, were used in the present study. These methods are implemented on a function-by-function basis in the Python modules skfeature (Li et al., 2018) and sklearn.

Classification algorithms

A classification algorithm has two phases: learning and classification. The classification model is trained on the given dataset and its label information during the learning phase; during the classification phase, the classification model assigns the label to the new dataset. The classification model in this paper uses LR (Han, Pei & Kamber, 2011) and SVM (Wang et al., 2018), both of which are classic binary classification models that are widely used in a variety of fields.

Leave-one-out validation

The dataset was divided into the training and validation sets. The training set was used to train the model, whereas the validation set was used to assess the model’s generalizability. If the size of the dataset $D$ was $N$, then $N-1$ pieces of data should be used for training, and the remaining data should be used for validation. A total of $N$ times are calculated for each group taken from D as the verification set until all samples have been verified as the set. Finally, the mean value of the verification error was calculated. This method is called leave-one-out cross-validation (Torgo, 2010).

Performance measures

The classification accuracy for each model was measured using the accuracy ratio (=(TP + TN)/(P + N)), precision ratio (=(TP)/(TP + FP)), and recall ratio (=TP/P), where P, N, TP, TN, FP, and FN are the positive, negative, true positive, true negative, false positive, and false negative prediction values in the confusion matrix, respectively.

Experimental studies

We obtained the ASV time series data of 25 vaginal communities. The ASV is regarded as the node of the network. The interaction between ASVs is used to construct the network edge, and the weight of the network edge was determined through the correlation coefficient between ASVs. Therefore, an ASV correlation coefficient network was constructed, which is an undirected and weighted network, and different labels (ABV, SBV, HEA) were assigned to the network according to the diagnosis of vaginal community. The correlation coefficient was calculated using the “spacc” function in the R package SpiecEasi, and the parameters in the “spacc” function were set as default values. Because there are too many zeros in the time series of some ASVs, in the process, we summed the counts of ASVs across the 25 datasets and selected ASVs with sum of counts in the top 60 to construct the networks. In 25 datasets, the counts sum of top 60 ASVs has proportion of 0.996 in the sum of all ASVs counts (Fig. S1). This ensures that the time series of ASVs contain many zero values cannot be selected. It is statistically meaningless to use them to construct network. The entire calculation process was performed using R-script (ASVCN.R). The weight of the edge of the network was extracted, and a vector (called network feature vector) was constructed to characterize the corresponding network. In total, we obtained 25 network feature vectors. Feature selection algorithms (DT and RF) were subsequently used to explore the network feature edge (python script: FeatureSelection.py) that plays an important role in the classification of vaginal community status.

In the present study, we aimed to determine the network feature edges that can distinguish SBV from ABV, ABV from HEA, SBV from HEA, and BV (SBV+ABV) from HEA. Therefore, the network feature vectors were categorized into four corresponding groups of data, and the grouped network labels were digitized. For example, the dataset of SBV vs ABV group only contained network feature vectors labeled SBV and ABV, and maps the SBV label to 1 and the ABV label to 0. In each grouping, owing to the small number of network feature vectors (SBV vs ABV: 21; ABV vs HEA:10; SBV vs HEA:19; BV vs HEA:25), we used the Leave-One-Out method to score the importance of network edges. For example, for grouping SBV vs ABV, each time one network feature vector was left, others 20 feature vectors were used as predictors to input the feature selection algorithm (DT and RF) which can generate the importance values of network edges. We repeated this process 21 times, ensured that each feature vector was left once and only once, then the whole process scored each network edge 21 times. Finally, the average of these scores was calculated as a measure of the importance of network edges, and the network edge was sorted according to size to facilitate the selection of network feature edges.

To verify whether the selected network feature edges can distinguish the community status and potentially be used as a marker for BV diagnosis, we selected the network feature edge as the predictors and verified it with the classification algorithm (SVM and LR). Specifically, according to the ranking results of the importance of network edges, the top k (=5, 10, 15, 20, and 25) network feature edges with high scores were selected as the predictors of the classification algorithm. Moreover, the corresponding label vector was used as the response vector of the classification algorithm to input the classification algorithm. During this process, we also used the Leave-One-Out method to randomly select a network feature vector as the test set and other network feature vectors as the training set. After the classifier training, we predicted the label of the test set. After implementing the Leave-One-Out method, the classification evaluation index was calculated according to the predicted value and actual value of the label of the network feature vector. We implemented the Leave-One-Out method 20 times and finally calculated the classification evaluation indictors, which reflects the overall ability of the selected network features to identify the states of the community.

BV vs HEA results

In Table 1, the feature selection algorithms investigated in this work obtained different results. They shared no common feature edges in their top 5 rankings.

Table 2 shows that LR performs better in DT feature edges than SVM; conversely, SVM performs better in RF feature edges than LR; the measures (ACC, Pre, and Recall) of LR with DT feature edges can achieve 1 in the top 5 and 10 feature edges.

SBV vs HEA results

Table 3 shows that feature selection algorithms also obtained different results from each other and did not share common feature edges in the top 5 rankings. The DT results show that the MIV (0.42) of the feature edge Lactobacillus.jensenii-Peptoniphilus.KQ960236_s is higher than that of other feature edges.

Table 4 shows that the performance of LR and SVM with DT results is better than that with RF results. Both the measures of LR and SVM achieve 1 in the top 5 feature edges of DT results. The measures of SVM also achieve 1 in the top 15 feature edges of DT results.

ABV vs HEA results

Table 5 shows that the feature selection algorithms also obtained different results from each other. Although there are no common feature edges in the two results, they have a common node (Lactobacillus.iners).

Table 6 shows that the measures of LR with DT results achieve 1 in the top five feature edges. The measures of SVM with DT results achieve 1 in the top 10 feature edges.

SBV vs ABV results

Table 7 shows that the feature edges in two results are different. The node (Lactobacillus.iners) appears in two results. The MIV of edge (JH591066_s-Gemella.haemolysans) is bigger than that of edges.

Table 8 shows that no measures of LR/DT, LR/RF, SVM/DT, and SVM/RF achieve 1. However, SVM/DT has better performance in the top 15 feature edges. The performances of the feature edges selected by the feature selection methods are not satisfactory.

Based on the abovementioned results, the following conclusions can be inferred:

• Machine learning can distinguish vaginal community states (BV, ABV, SBV, and HEA) based on a few feature edges (bacterial interaction). The classification performance between BV, SBV, ABV, and H is better than that between SBV and ABV.

• Selecting the top five feature edges of importance can achieve the best accuracy for the feature selection and classification model. In some cases, the more feature edges will decrease the performance of the classification algorithm.

• The feature edges selected by DT outperform those selected by RF in terms of classification algorithm LR and SVM. However, LR with DT feature edges is more suitable for diagnosing BV.

• The results of two feature selection algorithms exhibit differences in the importance of ranking of edges. Taking the top five feature edges as an example, the feature edges chosen by the two algorithms have almost no intersection.