An enhanced BERT model with improved local feature extraction and long-range dependency capture in promoter prediction for hearing loss

Data sources

In this study, we utilized datasets corresponding to three specific diseases: (1) Hearing loss, (2) breast cancer, and (3) cervical cancer. Hearing loss-related gene data were obtained from the following four databases: ClinVar (Landrum et al., 2020), DVD (Azaiez et al., 2018) (Deafness Variation Database), OMIM (https://www.omim.org), and HHL (Bolz, 2016) (Hereditary Hearing Loss). Breast cancer data were sourced from METABRIC (Mukherjee et al., 2018) (Molecular Taxonomy of Breast Cancer International Consortium), and cervical cancer data were obtained from CCDB (Agarwal et al., 2011) (Cancer of the Cervix Database). Gene entries from each database were extracted, deduplicated, and summarized; the number of genes sourced from each database is presented in Table 1.

Experimentally validated human promoter sequences were obtained from the EPDnew database. Based on the disease-related genes identified above, we subsequently matched the corresponding promoter sequences in EPDnew to construct the positive dataset. For the negative dataset, we first identified all experimentally validated human promoters within the genome and removed them, resulting in a genome-wide sequence dataset devoid of promoters (Umarov et al., 2019). From this dataset, we randomly selected contiguous genomic sequences matching the lengths of actual biological promoters, thereby generating the negative samples.

To assess the impact of sequence length on model performance, we considered two promoter sequence lengths: 300 base pairs (bp) (Wang et al., 2023) (covering positions −249 to +50 relative to the transcription start site, TSS) and 600 bp (Dogan et al., 2015) (spanning positions −499 to +100 relative to the TSS). Here, “+50” and “+100” indicate the number of base pairs downstream of the TSS. This approach acknowledges that promoter regions encompass not only upstream regulatory elements but also downstream regions that are critical for transcriptional regulation.

To evaluate the influence of different datasets on model prediction performance, we designed three fine-tuning strategies:

(1) Individual learning: Both model training and testing were conducted exclusively on the hearing loss dataset.

(2) Cross-disease training: Model training was performed using publicly available promoter datasets from breast cancer and cervical cancer, with testing conducted on the hearing loss dataset. This approach aims to leverage larger datasets from related diseases to enhance model performance when the hearing loss dataset is limited in size. Additionally, it allows an assessment of the model’s transferability and generalization across different disease contexts.

(3) Global training: Model training incorporated all known human promoter datasets, with testing performed on the hearing loss dataset. This strategy maximizes the available training data to achieve robust model learning.

The data distribution for training, validation, and testing under these three strategies is detailed in Table 2. Collectively, these experimental strategies facilitate a systematic evaluation of how disease-specific and cross-disease datasets affect the model’s ability to predict hearing-loss-related promoters, providing insights into its generalizability and robustness.

DNABERT-2 model

DNABERT-2 (Zhou et al., 2024) is an enhanced model built upon the widely used Transformer architecture (Vaswani et al., 2017), specifically developed for DNA sequence analysis. Building on the foundation of its predecessor DNABERT (Ji et al., 2021), which demonstrated notable performance in genomic tasks, DNABERT-2 introduces several key improvements. It incorporates byte pair encoding (BPE) (as illustrated in Fig. 1) as a replacement for the traditional k-mer method, thereby enhancing computational efficiency and enabling better sequence compression. Furthermore, Attention with Linear Biases (ALiBi) replaces the conventional positional embeddings, thus improving the model’s capability to handle longer sequences more effectively. Moreover, the integration of Flash Attention and the GeGLU (gated linear unit (GLU) and generalized linear unit (GELU)) activation function further enhances the model’s performance and computational resource efficiency. Collectively, these advancements enable DNABERT-2 to achieve superior accuracy and efficiency in a variety of genomic analysis tasks.

Feature extraction strategy via multi-model integration

In genomic sequence analysis, extracting comprehensive features from DNA sequences is essential to capture both global context and local sequence patterns. To address this challenge, we propose a multi-model integration strategy that leverages the strengths of different deep learning architectures.

BERT, based on a bidirectional encoder architecture, reads sequences from both left to right and right to left, making it highly effective for capturing contextual information. Moreover, BERT’s extensive pre-training on large-scale unlabeled data enables it to learn universal sequence representations, which can then be fine-tuned for various downstream tasks.

However, gene sequences typically contain a substantial number of local regulatory sites and short-range patterns that require specialized local feature analysis. CNNs, with their local convolutional operations, efficiently extract such features from sequences. Consequently, we initially combined BERT with CNNs so that CNNs could refine the local regions of the global features extracted by BERT. Through convolutional kernels and pooling operations, CNN is effective in capturing micro-patterns and local structures, which is crucial for identifying short-range dependencies within promoter sequences.

Furthermore, complex long-range dependencies play a critical role in gene regulation, especially through three-dimensional chromatin conformations. In this process, distal regulatory elements interact with promoters via chromatin loops, thereby influencing gene expression. To address these dependencies, we integrated BiLSTM, which has a bidirectional architecture that simultaneously captures both forward and backward dependencies, enabling comprehensive modeling of long-range relationships, particularly in recognizing distant regulatory elements.

As illustrated in Fig. 2, we constructed a multi-layer feature extraction architecture based on the DNABERT-2 model. Initially, BERT provides a global semantic representation through its contextual extraction capabilities. Subsequently, CNN refines these global features by capturing local patterns within sequences. BiLSTM is then applied to further integrate the features from both modules and to optimize long-range dependency modeling. This integration strategy merges the advantages of global context modeling and local structure extraction, thus forming a comprehensive multi-level feature representation system.

Promoter prediction using DNABERT-2 variants

The overall architecture of this study is illustrated in Fig. 3. All experiments were conducted on a Windows operating system with Python 3.10 as the programming environment. To accelerate training, an RTX 3090 GPU with 24 GB of VRAM was employed. All models were trained under identical hardware conditions to ensure fair comparisons. The complete source code is available at the following GitHub repository: https://github.com/Cqerliu/DNABERT_CBL.

To investigate the influence of different network architectures on model performance, we designed multiple experimental schemes and proposed two variant strategies aimed at optimizing feature extraction:

Single-module enhancement: DNABERT-2 + CNN and DNABERT-2 + BiLSTM.

This strategy individually augments DNABERT-2 with either a convolutional neural network or a bidirectional long short-term memory network after the BERT encoder. CNN modules effectively capture local patterns within the input sequences, while BiLSTM modules are well-suited for modeling long-range dependencies. This approach allows us to examine the isolated effects of each module on feature extraction and prediction performance.

Multi-module combination enhancement: DNABERT-2 + CNN + BiLSTM.

This strategy sequentially integrates both CNN and BiLSTM modules to form a comprehensive, multi-module network. Initially, the CNN extracts local features, which are then passed to the BiLSTM layer to model long-distance relationships. This hierarchical design leverages the strengths of both modules, aiming to maximize the overall predictive power.

The specific architectures and hyperparameters for each model are as follows:

(1) Baseline model (DNABERT-2_BASE): This model consists solely of the BERT encoder followed by a fully connected classification layer. The BERT backbone employed is ‘zhihan1996/DNABERT_2-117M,’ coupled with a single-layer fully connected classifier.

(2) Single-module model (DNABERT-2_CNN): A convolutional layer is added after the BERT output to enhance local feature extraction. The architecture is BERT → convolutional layer → max pooling layer → fully connected classifier. The convolutional layer has a kernel size of 3 and 1,536 output channels, with a pooling window size of 2. The ReLU activation function is used to introduce non-linearity.

(3) Single-module model (DNABERT-2_BiLSTM): A BiLSTM layer is added after the BERT output to capture long-range dependencies. The architecture is BERT → BiLSTM layer → fully connected classifier. The BiLSTM layer has 128 units and uses ReLU activation.

(4) Multi-module model (DNABERT-CBL): This model first applies a CNN layer to extract local features, followed by a BiLSTM layer to model long-distance relationships. The architecture is BERT → convolutional layer → max pooling layer → BiLSTM layer → fully connected classifier. The convolutional layer has a kernel size of 3 and 1,536 output channels, while the BiLSTM layer has 128 units. ReLU activation is consistently applied throughout the model.

All models were trained using the RTX 3090 GPU with 24 GB VRAM to ensure consistent training environments. The hyperparameters used in the experiments are summarized in Table 3. We employed the Adam optimizer with a learning rate of 3e−5 to avoid overfitting and ensure stable convergence. Models were trained for five epochs, with evaluation performed every 200 steps, and the best-performing model on the validation set was saved. A batch size of 8 was used during training to balance GPU memory utilization and computational efficiency, and increased to 16 during evaluation to speed up inference. ReLU activation was chosen throughout for its effectiveness in mitigating gradient vanishing and accelerating convergence. Given the binary classification nature of the task, the cross-entropy loss function was selected for its widespread applicability and proven performance. The total number of parameters for each model is provided in Table 4.

Evaluation metrics

To comprehensively evaluate the performance of each model, we adopted a suite of well-established classification metrics, including:

Loss: Measures the discrepancy between predicted outputs and true labels, providing insights into convergence and optimization quality. AUC: Assesses the model’s capability to distinguish between classes, with higher values indicating better discrimination. ACC: The proportion of correctly classified samples among all samples, reflecting overall classification performance. Matthew’s correlation coefficient (MCC): A robust metric that accounts for all elements of the confusion matrix, particularly suitable for imbalanced datasets. F1-score: The harmonic mean of precision and recall, balancing the trade-off between false positives and false negatives. Specificity (Sp): Measures the true negative rate, indicating how well the model identifies negative samples.

This combination of metrics offers a comprehensive perspective on each model’s strengths and weaknesses, enabling an in-depth analysis of their predictive performance.

Results of the cross-disease training strategy

In the cross-disease training strategy, the models were pre-trained on specific disease datasets (breast cancer and cervical cancer) before testing on the hearing loss dataset. For the breast cancer dataset, training and validation sets were split 8:2, followed by testing on the hearing loss dataset. The cervical cancer dataset was similarly processed. Tables 6 and 7, along with Figs. 5 and 6, display the performance metrics and comparisons. In both cases, DNABERT-2_C_A_BL consistently achieved the highest ACC and F1-scores, demonstrating strong generalization ability from breast and cervical cancer datasets to hearing loss promoters.

In cross-disease training, DNABERT-CBL also excelled. For example, with training on the breast cancer dataset, the variant with average pooling achieved an AUC of 0.955, compared to the baseline’s 0.930. Similarly, for the cervical cancer dataset, DNABERT-CBL achieved an AUC of 0.948 at 600 bp, surpassing the baseline’s 0.931. Figures 5 and 6 further validate the model’s advantages in transfer learning.

Results of the global training strategy

The global training strategy involved training on a dataset containing all known human promoter sequences, excluding hearing-loss-related data, and then testing on the hearing loss dataset. Promoter sequences associated with hearing loss were removed from the training data, which was then split 8:2 for training and validation. Table 8 provides the performance metrics, and Fig. 7 visually compares the models’ performance, showing that DNABERT-CBL consistently outperformed the baseline model in terms of stability and AUC across both sequence lengths.

Under the global training strategy, DNABERT-CBL again demonstrated strong performance, consistently achieving lower loss and higher accuracy across both sequence lengths. It also exhibited greater AUC stability, as shown in Fig. 7.

Comprehensive evaluation

Across all strategies, DNABERT-CBL demonstrated superior performance, particularly on large datasets and complex tasks. The AUC heatmap in Fig. 8 further illustrates its consistent advantage. DNABERT-CBL excelled in loss, accuracy, and AUC, outperforming the baseline and exhibiting robust adaptability to diverse sequence lengths and datasets.

In summary, the DNABERT-CBL model demonstrates outstanding performance in both individual learning and transfer learning, making it especially suitable for handling sequences of varying lengths and complex datasets. It exhibits strong adaptability and robustness across different conditions.

Comparison with other BERT-based methods for predicting promoters

Currently, few studies utilize BERT and its variants for promoter prediction. To evaluate our model’s effectiveness, we compared it with three state-of-the-art BERT-based methods: BERT-Promoter (Le et al., 2022), msBERT-Promoter (Li et al., 2024), and TSSNote-CyaPromBERT (Mai, Nguyen & Lee, 2022). All models were tested on the same benchmark dataset, splitting the hearing loss promoter sequences into training, validation, and test sets at a 6:2:2 ratio. Due to the 512-bp input length limitation of these models, only 300 bp experiments were conducted. Table 9 presents the performance metrics across all methods.

Our models, DNABERT-2_C_M_BL and DNABERT-2_C_A_BL, achieved the highest scores in accuracy, F1-score, Matthew’s correlation coefficient (MCC), and precision. Notably, DNABERT-2_C_A_BL with average pooling achieved an accuracy of 0.867, an F1-score of 0.866, an MCC of 0.742, and a precision of 0.875, demonstrating robust classification capability. Although its AUC (0.903) was slightly lower than that of msBERT-Promoter (0.939), this discrepancy may be attributed to a relatively weaker ability to discriminate borderline samples, thereby slightly reducing the area under the ROC curve. However, considering that AUC reflects an overall ranking performance, while DNABERT-2_C_A_BL achieved higher scores in core classification metrics such as accuracy and F1-score, this suggests that our model is more reliable and practically effective in binary classification tasks. Overall, the performance of DNABERT-2_C_A_BL still outperformed existing methods, affirming the model’s effectiveness and stability in promoter prediction.