Optimized Gaussian naïve Bayes for prostate cancer detection in White patients using gene expression data

Data gathering

This study utilized a dataset comprising gene expression and phenotype data for prostate cancer. The HTSeq—Counts gene expression and GDC TCGA-PRAD phenotype data were retrieved on August 25, 2024, from the UCSC Xena browser—(https://xenabrowser.net/) (Goldman et al., 2020). The dataset had been pre-normalized using log2(count+1) transformation to ensure the values were on a comparable scale.

Data preprocessing

During preprocessing, the gene expression and phenotype datasets were stratified by race. Python packages pandas (version 2.2.2) and NumPy (version 1.26.4) were employed to remove rows with missing values and to round the data from floating-point to integer types. Then, the processed dataset was labeled as “cancer” and “normal” using scikit-learn (version 1.5.1).

Feature selection

DEG analysis was conducted on the labeled data using the PyDESeq2 library (version 0.4.10). PyDESeq2 was chosen for its robustness in identifying differentially expressed genes through effective statistical methods. The DEG results served as features for ML model development. Features were selected based on specific thresholds, as outlined in Table 1. Each parameter plays a distinct role in feature selection: basemean represents the average normalized expression level, log2FoldChange determines upregulated and downregulated genes, and padj identifies statistically significant genes (Love, Huber & Anders, 2014). A positive log2 foldchange value indicates upregulated with higher expression, whereas a negative value indicates downregulated genes with lower expression. This metric is essential for highlighting biologically relevant changes in gene expression. The log2FoldChange thresholds (0.35 and 0.4) were selected to investigate their impact on model performance and are commonly applied in DEG studies to capture biologically relevant expression variations.

Moreover, receiver operating characteristic (ROC) analysis was performed to identify potential biomarker genes. An area under the curve (AUC-ROC) value closer to 1 indicates superior discrimination, with high true-positive sensitivity and low false-positive rates (Hajian-Tilaki, 2013). This study applied an AUC-ROC threshold greater than 0.9, indicating excellent discriminatory power (Casarrubios et al., 2022; Unal, 2017). ROC visualizations were generated using Matplotlib (version 3.8.3). As gene identifiers were in Ensembl_ID, the SynGO website was used to convert them to gene name symbols (Koopmans et al., 2019). The selected genes were further verified via Gene Set Enrichment Analysis (GSEA) using the Molecular Signatures Database (MsigDB) to confirm their association with prostate cancer (Liberzon et al., 2015).

Gaussian naïve Bayes modeling with several scenarios

The dataset was divided into training and test sets using split ratios of 60/40, 70/30, and 80/20. Class imbalances were mitigated by applying balancing techniques to augment the minority class, achieving a 1:3 ratio. To maintain the natural distribution and ensure unbiased evaluation, balancing was applied solely to the training data; the test data remained unmodified. The balancing methods included Random Oversampling, KMeans SMOTE, Borderline SMOTE and SMOTEEN. SMOTE-based methods were prioritized for their ability to generate synthetic samples with high-dimensional genomic data. Adaptive Synthetic Sampling (ADASYN) was excluded, as it may over generate synthetic instances for noisy or ambiguous samples, potentially comprising classifier robustness in small biomedical datasets (Mitra, Bajpai & Biswas, 2023). Balancing was implemented using imbalanced-learn package (version 0.12.3). Models were further optimized through hyperparameter tuning.

This study explored multiple scenarios incorporating various feature selection methods, data split ratios, balancing techniques, and hyperparameters, detailed in Table 2, yielding a total of 96 scenarios.

Model evaluation

Model performance was evaluated through classification reports, encompassing metrics such as accuracy, precision, recall, and F1-scores. These metrics are crucial for evaluating the model harmonization between training and test data. Accuracy reflects the percentage of correctly classified samples (cancer or normal), precision measures the percentage of true positives among predicted positives, recall indicates the percentage of actual positive correctly identified; and the F1-score represents the harmonic mean of precision and recall (Miller et al., 2024). To assess overfitting or underfitting, performance metrics (accuracy, precision, recall and F1-score) were compared between training and test sets, with consistency suggesting strong generalization. Model evaluation was performed using scikit-learn (version 1.5.1). Following classification, the best-performing model was interpreted using LIME to identify the most influential genes contributing to individual predictions.

Black race validation

To evaluate the model’s generalizability across racial groups, validation was performed on the Black race dataset, which had been separated during initial preprocessing steps. The identical preprocessing pipeline was applied, including data transformation and feature selection using DESeq2 with the same thresholds. The model, trained on the White race dataset, was applied without retraining to classify the Black race test data, enabling a direct comparison of cross-racial performance.

Dataset gathering

The dataset used in this study comprised HTSeq-Counts gene expression data and GDC TCGA-PRAD Phenotype data. The HTSeq—Counts dataset included 60,488 genes and 551 patient samples, while the GDC TCGA-PRAD Phenotypes dataset contained 172 medical record variables. This study focused on the White population for building the ML model for prostate cancer detection, as it constituted the majority of the dataset (approximately 83%, or 458 of 551), providing a robust foundation for training process. Additionally, the Black race subset was used to validate the model’s performance. The distribution of age and Gleason score for White patients is depicted in Fig. 2. The age distribution indicates that the majority of patients were around 62 years old, suggesting a predominance of older individuals in the White race cohort. Additionally, the Gleason score distribution shows that a score of 7.0 was the most prevalent. Normal Gleason score range between 2–5, whereas scores of 6–10 indicate abnormal prostate cancer cell; higher scores reflect greater cellular abnormality (Tagai et al., 2019). The corresponding distributions for Black patients are shown in Fig. 3.

Data preprocessing

First, the data was cleaned by removing missing values and rounding the values into integers. As a result, the gene count was reduced from 60,488 to 57,429. Samples were then labeled as “cancer” or “normal” based on the TCGA barcode, where cancer is “01” denoted cancer and “11” indicated normal. The labeled samples were stored in a metadata for subsequent analyses, including feature selection, model training, and performance validation.

Feature selection

Four DEG analysis scenarios were conducted resulting 139, 7, 4 and 13 genes for scenario 1, 2, 3, and 4, respectively as detailed in Table 3.

To refine scenario 1 further, the 139 genes were subjected to ROC analysis, resulting in a reduced set of 13 genes. The ROC analysis demonstrated high sensitivity for these 13 genes in detecting prostate cancer, as illustrated in Fig. 4.

The Ensembl IDs of these 13 genes were converted to gene symbols and cross-referenced with MSigDB to assess their correlation with prostate cancer. According to MSigDB, nine out of the 13 genes, including GJA1, CLU, HSPB8, MYLK, CRYAB, FGFR2, GSTM3, GSTP1, and FRMD6, exhibited association with prostate cancer, as shown in Fig. 5. These genes suggest involvement in cancer-related pathways. The remaining genes (FAM107A, FAXDC2, SCARA3 and APOBEC3C) showed no direct correlation with prostate cancer. Despite this, these four genes were retained due to their strong predictive performance in ROC analysis, with AUC-ROC scores exceeding 0.9, indicating excellent discriminatory ability between cancer and non-cancer cases.

Gaussian naïve Bayes Modeling

The gene sets from each scenario were processed using various splitting ratios and balancing techniques. The result for all modeling scenarios are presented in Tables 4–7.

As shown in Table 4, the optimal performance for prostate cancer detection was achieved with a 60/40 splitting ratio and the Random Oversampling, resulting a test accuracy of 92% for White patients and 93% for Black patients. This results also maintained high precision, recall, and F1-score values across both racial groups, reaching up to 96% for cancer classification. The effectiveness of only four features highlights the strength of the feature selection process, demonstrating that a minimal gene set can produce robust and reliable classification results. Furthermore, applying hyperparameter tuning did not significantly alter outcomes, indicating consistent model performance under default settings.

With 13 features (Table 5), the highest test accuracy of 95% for White patients and 93% for Black patients was attained using a 70/30 splitting ratio and KMeansSMOTE. This setup delivered consistent performance across metrics, with cancer detection scores up to 98%. Hyperparameter tuning had minimal impact, indicating the model’s robustness. Compared to other scenarios, this setup achieved a better balance between accuracy and generalizability across racial groups while utilizing a slightly larger feature set. The no-balancing scenario was only applied to the top performing datasets to ensure fair assessment without altering sample distributions. Accuracies without balancing were lower than those with balancing techniques, indicating that a moderate gene set, combined with suitable balancing and splitting, enhances accuracy and generalizability in race-aware prostate cancer classification models.

For 139 features (Table 6), the highest test accuracy of 94% with a 70/30 splitting ratio and Random Oversampling. This configuration consistently yielded strong cross-racial performance, with cancer detection metrics up to 99% in both groups. Random Oversampling proved most effective for this larger feature set, consistently appearing in top scenarios and providing stability.

Using seven features (Table 7), the model achieved its best test accuracy of 93% across racial groups with KMeansSMOTE. Performance was stable regardless of hyperparameter tuning or train-test split ratios (80/20, 70/30, and 60/40), with cancer metrics up to 97%. This highlights KMeansSMOTE’s reliability for smaller feature sets in race-specific cancer classification.

Among the 96 scenarios, the Gaussian naïve Bayes algorithm achieved its highest test accuracy of 95% for the White population and 93% for the Black population using 13 genes, a 70/30 split, the KMeansSMOTE, and hyperparameter tuning. This configuration not only produced excellent accuracy but also maintained strong precision, recall, and F1-scores, particularly in cancer classification across racial groups.

To enhance interpretability, LIME was applied to the best-performing Gaussian naïve Bayes model to identify key genes influencing classification decisions. LIME visualization highlighted impactful genes for individual predictions, as shown in Fig. 6. For a correctly predicted cancer case, high-weighted genes such as ENSG00000152661.7 (expression: 12.00) and ENSG00000109846.6 (expression: 11.00) positively contributed to the cancer classification. Although the normal class was mispredicted as cancer (label 0) in one instance, LIME confirmed biological relevance, aligning with gene expression patterns.

In Fig. 6, each gene’s expression reflects its activity value in the sample (e.g., ENSG00000084207.14 at 12.00 was more active than ENSG00000168309.15 at 9.00). Blue colored genes supported cancer predictions, while orange colored favored normal. This visualization supports model reliability, amid class imbalances. Interpretability focuses on the 13-gene dataset for clarity on key biological signals. Figure 7 ranks the top 13 gene by LIME importance, with higher bars indicating greater influence.

Model evaluation

This study optimized Gaussian naïve Bayes to achieve 95% accuracy. In comparison, a prior study using AdaBoost with feature selection techniques such as Sequential Floating Forward Selection (SFFS), RELIEFF, Sequential Backward Feature Section (SFBS) and Significant Attribute Evaluation (SAE) reported a 91.1% accuracy (Tirumala & Narayanan, 2019). This comparison highlights Gaussian naïve Bayes’ potential for gene expression datasets.