Linear B-cell epitope prediction for SARS and COVID-19 vaccine design: Integrating balanced ensemble learning models and resampling strategies

Dataset description

The empirical dataset used in this study is the B-cell epitope dataset, developed during research into epitope prediction and vaccine design (Noumi et al., 2021). The dataset is publicly available on Kaggle, thereby promoting reproducibility and transparency in scientific research (Noumi et al., 2021; Kaggle, 2024). This dataset provides detailed information about peptides and their corresponding proteins, alongside physicochemical features relevant for epitope prediction (Jain et al., 2021; Noumi et al., 2021). The dataset is derived from IEDB and UniProt. It contains 14,387 rows representing all combinations of 14,362 unique peptides and 757 proteins (Kaggle, 2024). The dataset includes 13 independent variables (features) and 1 dependent variable (target). The dataset includes peptide-level features such as the start and end positions that define the peptide’s location within the parent protein, and the amino acid sequence of the peptide. It also includes physicochemical properties like chou_fasman (β-turn propensity), emini (relative surface accessibility), kolaskar_tongaonkar (antigenicity), and parker (hydrophobicity). In terms of protein-level features, the dataset provides the parent protein’s identifier and its amino acid sequence, as well as properties such as isoelectric point, aromaticity, hydrophobicity, and stability (El-Manzalawy, Dobbs & Honavar, 2017; Noumi et al., 2021; Ozger & Cihan, 2022). For this experimental dataset, the target variable indicates the binary classification of antibody valence, distinguishing between positive and negative classes. Specifically, the dataset contains 10,485 rows labeled as negative (0) and 3,902 rows labeled as positive (1), highlighting a significant class imbalance (Kaggle, 2024). This imbalance can negatively impact the performance of machine learning models by biasing predictions toward the majority class (Khushi et al., 2021; Kraiem, Sánchez-Hernández & Moreno-García, 2021; Wang & Cheng, 2021; Gurcan & Soylu, 2024a). To address this issue, resampling techniques were employed to balance the dataset, ensuring that the model can learn effectively from both classes and improve its generalization capability (Gurcan & Soylu, 2024b).

Feature engineering and data preprocessing

Robust data preprocessing (cleaning, normalization) and feature engineering are essential for optimizing dataset usability and predictive accuracy in machine learning (Haixiang et al., 2017; Plotnikova, Dumas & Milani, 2020; Ahsan et al., 2021). To enhance the predictive capability of the model, several additional features were derived from the original dataset. The protein sequence length was calculated as the length of the protein_seq column, while the peptide sequence length was determined as the length of the peptide_seq column. The parent protein ID length was obtained as the length of the parent_protein_id column. Additionally, the peptide length was computed as the difference between the end_position and start_position columns, plus one (Caoili, 2022). To ensure that categorical variables could be effectively used in machine learning models, all string-based categorical features were converted into numerical format using label encoding. Specifically, the columns “parent_protein_id”, “peptide_seq”, and “protein_seq” were encoded as integers to enable compatibility with the classification models (Raschka & Mirjalili, 2019; Raschka, Patterson & Nolet, 2020; Gurcan, 2023; Gurcan & Soylu, 2024b). These engineered features captured structural and positional variations in the peptide and protein sequences, enriching the feature space for model training (Jespersen et al., 2019; Amrun et al., 2020; Bukhari et al., 2022). As a result, the total number of features increased from 14 to 18, providing the model with a more comprehensive representation of the data (Jespersen et al., 2019; Ozger & Cihan, 2022; Gurcan, 2023).

After new features were incorporated into the dataset, recursive feature elimination (RFE) was employed to identify the most relevant features for B-cell epitope prediction. RFE is a feature selection method that iteratively removes less important features to select those that contribute most significantly to the model’s predictive performance. The RFE algorithm was implemented using the Random Forest classifier as the base estimator. Feature selection was performed on the original (baseline) dataset without applying any resampling techniques. The optimal number of features was determined based on the ROC AUC scores calculated for various feature subsets. At the end of the RFE process, all variables were ranked by their importance, and the subset of features that yielded the highest ROC AUC score was selected as the optimal feature set.

The relationship between the ROC AUC score and the number of selected features is visualized in Fig. 1. As shown in the figure, the highest performance was achieved using nine features. The top nine selected features identified by RFE are as follows: “parent_protein_id”, “peptide_seq_length”, “hydrophobicity”, “isoelectric_point”, “emini”, “parker”, “end_position”, “start_position”, and “protein_seq_length”. The eight features eliminated by the RFE algorithm were: “chou_fasman”, “aromaticity”, “protein_seq”, “kolaskar_tongaonkar”, “stability”, “peptide_seq”, “peptide_length”, and “parent_protein_id_length”. This process identified the most informative features and optimized their number, resulting in a refined feature matrix X for robust model performance.

The target column, target, was retained as the response variable. To standardize the feature space and accommodate varying scales among variables, Min-Max Scaling was applied, ensuring that all features are transformed into a common range between 0 and 1 (Haixiang et al., 2017; Ahsan et al., 2021; Gurcan, 2024; Scikit-learn, 2024). Additionally, to address the class imbalance in the target variable, several resampling techniques were applied, including oversampling the minority class and undersampling the majority class (Kraiem, Sánchez-Hernández & Moreno-García, 2021; Wang, Wang & He, 2023; Gurcan & Soylu, 2024a). Furthermore, hybrid-sampling approaches were utilized, combining both oversampling and undersampling methods (Khushi et al., 2021; Palli et al., 2022). The dataset was further divided into training and validation sets to ensure robust evaluation of the model. The train_test_split function was utilized to split the resampled data into training and testing subsets, with 80% of the data used for training and 20% reserved for testing. Stratified sampling (stratify=y_resampled) was employed during the split to preserve the class distribution in the target variable across both subsets, ensuring a balanced representation of classes (Pedregosa et al., 2011; Raschka & Mirjalili, 2019; Scikit-learn, 2024).

Application of resampling techniques

Resampling techniques are crucial in machine learning for addressing class imbalance, ensuring models perform well on minority classes (Wang et al., 2020; Gurcan & Soylu, 2024b). This is particularly important in B-cell epitope prediction, where balanced data improves classifier accuracy in identifying antibody-inducing regions (Ozger & Cihan, 2022). In this study, we implement eight resampling methods to address class imbalance, categorized into three types: over-sampling, under-sampling, and hybrid-sampling, along with a Baseline (no resampling) scenario for comparison (Khushi et al., 2021; Palli et al., 2022; Gurcan & Soylu, 2024a). Over-sampling methods aim to increase the representation of the minority class and include KM-SMOTE, which uses k-means clustering to generate synthetic samples more effectively (Xu et al., 2021; Gurcan & Soylu, 2024a), Synthetic Minority Oversampling Technique (SMOTE), which creates new samples by interpolating between existing ones (Zheng, Cai & Li, 2015; Kraiem, Sánchez-Hernández & Moreno-García, 2021), and Random Oversampling (ROS), which simply duplicates samples from the minority class (Khushi et al., 2021; Kraiem, Sánchez-Hernández & Moreno-García, 2021; Gurcan & Soylu, 2024a).

Under-sampling methods, designed to reduce the majority class representation, include IHT, which removes samples based on their classification difficulty (Wang, Wang & He, 2023; Gurcan & Soylu, 2024a), RENN, which eliminates majority samples misclassified by their nearest neighbors (Kim & Hwang, 2022; Gurcan & Soylu, 2024a), and ENN (edited nearest neighbor), which removes conflicting majority samples to improve class separation (Kraiem, Sánchez-Hernández & Moreno-García, 2021; Gurcan & Soylu, 2024a). Hybrid-sampling approaches integrate the strengths of both over-sampling and under-sampling methods to achieve better balance and improve model performance. Hybrid-sampling combines both strategies for greater balance, with SMOTE-ENN using SMOTE for generating synthetic samples and ENN for refining the majority class (Fotouhi, Asadi & Kattan, 2019; Kraiem, Sánchez-Hernández & Moreno-García, 2021), and SMOTE-TL, which integrates SMOTE with Tomek Link removal to enhance class boundaries (Khushi et al., 2021; Swana, Doorsamy & Bokoro, 2022). The Baseline scenario, which does not apply any resampling, serves as a reference point to assess the impact of these methods on model performance (Gurcan & Soylu, 2024a, 2024b). In summary, Table 1 provides an overview of how each resampling technique modifies the class distribution and addresses the imbalance ratio, highlighting their adjustments to the dataset.

The Baseline model, which does not apply any resampling, maintains the original class imbalance with a ratio of 2.69. Over-sampling techniques, such as SMOTE, K-Means Synthetic Minority Oversampling Technique (KM-SMOTE), and ROS, effectively balance the classes by increasing the minority class count to match the majority, achieving a ratio of 1.0. Under-sampling methods, including ENN, RENN, and IHT, reduce the majority class size, with IHT achieving the lowest imbalance ratio of 1.16. Hybrid approaches, such as SMOTE-ENN and SMOTE-TL, combine over- and under-sampling to adjust class distributions, with SMOTE-ENN slightly favoring the minority class and achieving a ratio of 0.94, while SMOTE-TL fully balances the dataset with a ratio of 1.0. These resampling strategies highlight their diverse impacts on class balance, shaping the dataset’s distribution and enhancing its suitability for effective model training (Kraiem, Sánchez-Hernández & Moreno-García, 2021; Wang, Wang & He, 2023; Gurcan & Soylu, 2024a).

Application of ensemble classifiers with hyperparameter optimization

In this study on B-cell epitope prediction, we employed ten ensemble models across three categories: Bagging, Boosting, and Balancing (Wang & Cheng, 2021; Gurcan & Soylu, 2024b, 2024a). Each category represents a distinct ensemble learning approach, designed to enhance model performance and address challenges such as class imbalance. Bagging, or Bootstrap Aggregating, reduces variance by training multiple models on random subsets of the data and aggregating their predictions (Gurcan & Soylu, 2024b, 2024a). Random Forest builds multiple decision trees and combines their outputs through majority voting, offering robustness against overfitting. ExtraTrees is similar to Random Forest but introduces randomness in feature splits to improve computational efficiency (Ahsan et al., 2021; Gurcan, 2024). The Bagging Classifier serves as a general-purpose ensemble method that combines predictions from base models trained on bootstrap samples (Plaia et al., 2022). Boosting focuses on improving model accuracy by sequentially training weak learners, with each iteration correcting the errors of the previous model (Al-Azzam & Shatnawi, 2021; Wang, Wang & He, 2023; Gurcan & Soylu, 2024b). Extreme gradient boosting (XGBoost) is an efficient gradient boosting framework optimized for performance on structured data. CatBoost specializes in handling categorical features with minimal preprocessing while maintaining high accuracy. Light Gradient-Boosting Machine (LightGBM) uses a leaf-wise tree growth strategy, enabling faster training and better scalability for large datasets (Gurcan & Soylu, 2024a, 2024b).

Balancing integrates resampling strategies within ensemble methods to address class imbalance, ensuring fair treatment of both classes (Khushi et al., 2021; Gurcan & Soylu, 2024a, 2024b). Balanced Random Forest (BalancedRF) incorporates under-sampling of the majority class during tree construction to improve minority class representation (Su et al., 2015). EasyEnsemble generates multiple balanced subsets of the majority class, training a separate model on each to mitigate bias (Wang & Cheng, 2021). Balanced Bagging (BalancedBG) combines under-sampling with bagging techniques to create diverse and unbiased ensembles (Khushi et al., 2021; Gurcan & Soylu, 2024a). Finally, Random under-sampling integrated in the learning of AdaBoost (RUSBoost) blends under-sampling with boosting, focusing on improving predictions for the minority class (Gurcan & Soylu, 2024a).

In an effort to enhance the performance of B-cell epitope prediction models, hyperparameter optimization for the ensemble classifiers used in this study was performed using the GridSearchCV technique. This method is a widely used, systematic, and exhaustive search method for tuning hyperparameters in machine learning models. It evaluates all possible combinations within predefined parameter ranges to identify the optimal set of hyperparameters for each model. During the optimization process, the ROC AUC score was used as the evaluation metric. This approach helps to reduce the risk of overfitting while enhancing the generalization ability of the models. The parameter grids for each classifier were determined based on prior literature and preliminary analyses, and a five-fold cross-validation (5-fold CV) was applied to ensure the reliability of the results. The best parameters obtained via GridSearchCV are presented in Table 2 and were used in the final model evaluations. Combining these approaches with fine-tuned hyperparameters enables the development of robust and balanced ensemble-based predictions for B-cell epitope identification.

Experimental setup and performance evaluation

The experimental setup for this study was conducted in a high-performance computing environment to ensure efficiency and stability. The system featured an Intel i7-12650H processor, Nvidia RTX 3060 GPU, and 16 GB of RAM, operating on a 64-bit version of Windows 10. Python 3.12, along with Jupyter Notebook, was utilized for the development and execution of the analysis (Pedregosa et al., 2011; Nelli, 2023). The selected tools and libraries ensured smooth execution throughout the machine learning pipeline, covering data preprocessing, model training, evaluation, and deployment. Python’s versatility and extensive ecosystem made it the perfect choice for this research, enabling efficient handling of complex tasks in machine learning and data science (Raschka, Patterson & Nolet, 2020; Gurcan, 2023; Nelli, 2023; Gurcan & Soylu, 2024a).

In this study, the performance evaluation was based on seven key metrics: Accuracy, Precision, Recall, F1-score, PR AUC, MCC, and ROC AUC (Gurcan & Soylu, 2024b; Disci, Gurcan & Soylu, 2025). Using these seven metrics allows for a comprehensive evaluation, ensuring that various aspects of model performance are considered, particularly in the presence of class imbalance (Nelli, 2023; Gurcan & Soylu, 2024a; Scikit-learn, 2024). Accuracy measures the overall proportion of correct predictions made by the model. Precision indicates the proportion of true positive predictions among all positive predictions (Kim & Hwang, 2022; Gurcan & Soylu, 2024b). Recall represents the proportion of true positives identified out of all actual positive instances (Kim & Hwang, 2022; Kaya & Gürsoy, 2023). F1-score provides a balanced measure between precision and recall, especially in cases of imbalanced classes (Al-Azzam & Shatnawi, 2021; Kaya & Gürsoy, 2023). PR AUC measures model performance under class imbalance by evaluating precision-recall trade-offs across thresholds. MCC evaluates classification quality by considering all confusion matrix categories, making it robust for imbalanced datasets. Finally, ROC AUC evaluates the model’s ability to discriminate between positive and negative classes, with higher values indicating better classification performance (Kraiem, Sánchez-Hernández & Moreno-García, 2021; Gurcan & Soylu, 2024b). This metric evaluates how well the classifier can separate the two classes across various threshold levels, making it a reliable indicator of model performance, even when the class distribution is uneven (Walsh & Tardy, 2023; Gurcan & Soylu, 2024b). Focusing on ROC AUC ensures that the models are not biased toward the majority class and can effectively identify rare events, such as the presence of antibodies in B-cell epitope prediction (Noumi et al., 2021; Walsh & Tardy, 2023; Gurcan & Soylu, 2024a). Finally, to assess whether the observed performance improvements between models resulted from true model enhancements rather than random variation or changes in data structure, three statistical significance tests were applied to the obtained results. Based on the ROC AUC scores derived from five-fold cross-validation, three widely accepted statistical tests, including the paired t-test, the Wilcoxon signed-rank test, and the permutation test, were conducted. Furthermore, confidence-based analyses were conducted to identify peptides most confidently and correctly predicted as potential epitope candidates, while high-confidence misclassifications were examined solely to assess model limitations. In the results section, all classifier performance tables, including those for resampling methods, are organized and presented in descending order based on ROC AUC scores to provide a clear and consistent comparison of model performance.

Evaluation of classifiers without resampling strategies

In this analysis, classifiers are evaluated in the baseline scenario without any resampling methods applied, focusing on various performance metrics. The metrics include: Accuracy, Precision, Recall, F1-score, PR AUC, MCC, and ROC AUC. Table 3 presents the performance metrics of classifiers in the baseline scenario, organized from highest to lowest ROC AUC scores, showcasing each model’s ability to discriminate between positive and negative classes in the B-cell epitope prediction task. When evaluating ROC AUC performance, BalancedRF (0.8640) emerges as the top-performing classifier, followed closely by ExtraTrees (0.8572) and BalancedBG (0.8560). Random Forest (0.8529) also demonstrates strong discriminative capability, while bagging-based methods (e.g., Bagging: 0.8312) consistently outperform boosting approaches like LightGBM (0.8199), XGBoost (0.8144), and CatBoost (0.7896). Hybrid balancing techniques such as RUSBoost (0.7445) and EasyEnsemble (0.7131) lag behind, highlighting the superiority of bagging with intrinsic class balancing (e.g., BalancedRF) for imbalanced classification tasks.

Evaluation of classifiers for over-sampling strategies

This analysis evaluates the performance of ten classifiers applied with three over-sampling methods (ROS, SMOTE, and KM-SMOTE) for B-cell epitope classification, focusing on their ability to address class imbalance effectively. Table 4 presents the top five classifiers for each oversampling method, ranked by multiple performance metrics and systematically ordered by descending ROC AUC values. In the ROS model, classifiers achieved the highest overall ROC AUC scores, with ExtraTrees (0.9466) and BalancedRF (0.9404) demonstrating the best performance. The KM-SMOTE model showed slightly lower ROC AUC values compared to ROS, where Extra Trees (0.9240) and Random Forest (0.9213) were the top performers. SMOTE provided a balanced but comparatively lower performance, with ExtraTrees (0.9163) and Random Forest (0.9139) achieving the highest ROC AUC scores in this category. As a result, the ROS model, particularly when paired with ExtraTrees, BalancedRF, and Random Forest classifiers, proved to be the most effective resampling approach for improving class discrimination.

Evaluation of classifiers for under-sampling strategies

This analysis examines the performance of ten classifiers applied with three under-sampling methods (IHT, RENN, and ENN) for B-cell epitope classification, focusing on their effectiveness in managing class imbalance. Table 5 displays the top five classifiers for each under-sampling technique, ranked by multiple performance metrics and ordered by descending ROC AUC values to highlight their classification performance. In the IHT model, classifiers achieved the highest overall ROC AUC scores among the under-sampling methods, with ExtraTrees (0.9799) and Random Forest (0.9787) leading the performance. The RENN model demonstrated slightly lower ROC AUC values compared to IHT, with ExtraTrees (0.9695) and BalancedRF (0.9595) as the top performers. The ENN model showed a balanced yet relatively lower performance, where ExtraTrees (0.9509) and BalancedRF (0.9486) ranked highest in ROC AUC. In summary, the IHT model, particularly in combination with ExtraTrees, Random Forest, and BalancedRF classifiers, emerged as the most effective under-sampling approach to improve classification performance.

Evaluation of classifiers for hybrid-sampling strategies

This analysis evaluates the performance of ten classifiers applied with two hybrid-sampling methods (SMOTE-ENN and SMOTE-TL) for B-cell epitope classification, emphasizing their ability to manage class imbalance. Table 6 presents the top five classifiers for each hybrid-sampling model, ranked based on multiple performance metrics and organized in descending order of ROC AUC. In the SMOTE-ENN model, classifiers achieved the highest overall ROC AUC scores among the hybrid-sampling methods, with Extra Trees (0.9899) and Random Forest (0.9856) demonstrating superior performance. The SMOTE-TL model showed slightly lower ROC AUC values compared to SMOTE-ENN, where ExtraTrees (0.9363) and BalancedRF (0.9318) were the top performers. Experimental results identified SMOTE-ENN as the most effective hybrid-sampling technique, particularly when integrated with ExtraTrees, Random Forest, and BalancedRF classifiers.

Overall comparative evaluation resampling models and ensemble classifiers

This comparative analysis presents a comparison of the ROC AUC scores for eight resampling methods and the baseline (no resampling) across ten classifiers, evaluating the impact of different resampling techniques on classifier performance, as shown in Table 7. In the table, the resampling methods and classifiers are sorted by their mean ROC AUC scores in descending order, which are shown in the last column and row. Additionally, in order to make the most successful combinations of resampling techniques and classifiers clearer, Fig. 2 illustrates their ROC-AUC scores for the top ten combined models, providing a visual representation of their performance.

The results highlight that SMOTE-ENN generally produces the highest performance across the classifiers, with ExtraTrees achieving the highest ROC AUC score of 0.9899. IHT follows closely behind, with ExtraTrees again performing the best at 0.9799. This indicates that IHT and SMOTE-ENN are highly effective resampling methods, particularly for ExtraTrees, BalancedRF, and Random Forest. These classifiers maintain strong ROC AUC scores across all resampling methods. In contrast, RUSBoost and EasyEnsemble demonstrated significantly lower ROC AUC scores across all resampling methods, with the weakest performances observed in the SMOTE and Baseline configurations. In summary, SMOTE-ENN, IHT, and RENN methods demonstrated superior classification performance, achieving the highest ROC AUC scores when paired with tree-based ensemble classifiers including ExtraTrees, BalancedRF, and Random Forest.

On the other hand, the Baseline (no resampling) consistently shows the lowest scores across all classifiers, highlighting the importance of resampling methods. For instance, ROC AUC scores improve significantly from 0.8143 (Baseline) to 0.9516 with IHT, as shown in the mean values (see Table 7), demonstrating the effectiveness of resampling in enhancing classifier performance.

Confidence-based evaluation of high-priority B-cell epitope candidates

In this phase, the model was evaluated on the validation set using correct predictions; misclassifications were examined to identify limitations. To provide a more comprehensive evaluation of the model’s behavior, representative prediction examples with the highest confidence scores for correct and incorrect classifications are presented in Table 8.

Peptides such as “SVATVQLP”, “PVMFVSRVPAKKPRK”, and “TVRHRCQAIRKKPLP” were correctly classified with 100% confidence and may serve as candidate epitopes, subject to experimental validation. In contrast, peptides like “KNVSPEARHPLV” and “LFEDAYLLTLHAGRV” were misclassified with high confidence, highlighting potential model limitations and complexity in peptide feature representation.

Peptides that were correctly classified with high confidence represent sequences that the model consistently identifies as likely B-cell epitopes. While these peptides may hold immunogenic potential, such properties must be confirmed through biological validation. Their ability to elicit antibody responses makes them particularly valuable for immunological studies. Misclassified high-confidence peptides reveal model limitations and potential complexities in the sequence data, such as structural features or mutations. These observations highlight which peptide types the model handles well or struggles with, and provide direction for model refinement and future experimental studies. Ultimately, biological validation remains essential to confirm the functional relevance of all predicted candidates.

Statistical significance tests

To confirm that the observed improvements in classification performance were not due to random variation or changes in data structure, various paired statistical tests were applied. Specifically, we conducted three widely accepted statistical tests based on the ROC AUC scores obtained from five-fold cross-validation. These included the paired t-test, the Wilcoxon signed-rank test, and the permutation test. The paired t-test evaluates whether there is a statistically significant difference between the mean performance scores of two models. The Wilcoxon signed-rank test, a non-parametric method, assesses median differences between paired distributions. The permutation test estimates the p-value by repeatedly permuting the labels of the data to determine whether the observed differences could have occurred by chance. For example, when comparing the Baseline and SMOTE-ENN resampling methods using the Random Forest classifier, statistically significant differences were observed in all tests. In general, performance improvements obtained through resampling methods were found to be significantly better than those of the baseline scenario. The average ROC AUC difference between the baseline and resampling techniques was calculated as 0.058290. The paired t-test yielded a p-value of 0.000086, the Wilcoxon test yielded 0.015620, and the permutation test yielded 0.015050. Since all p-values are below the 0.05 threshold, the differences can be considered statistically significant. These findings confirm that the performance gains achieved through resampling methods are not merely due to random variations, but rather reflect statistically significant model improvements.