Predicting amyloid proteins using attention-based long short-term memory

Introduction

In many organs and tissues, amyloid proteins tend to flock to create insoluble aggregates of different sizes. These amyloid aggregates can continuously build up to produce intracellular protein inclusions or extracellular plaques, most notably as a component of disease processes. They are mostly made up of $\beta$-sheet structures and have a fibrillary shape when aggregated (Rambaran & Serpell, 2008). Disease-related amyloid proteins, found as plaques inside the central nervous system, include those involved in the pathological trigger or growth of Alzheimer’s disease (AD). Amyloid-beta (A $\beta$) plaques, neurofibrillary tangles, and brain shrinkage are the hallmarks of AD, a neurodegenerative condition. It causes dementia most often (Beach et al., 2012; Matsui et al., 2008) and has a major societal effect (Nichols et al., 2019; Matsui et al., 2008). However, since AD’s clinical symptoms might overlap with those of other illnesses, including frontotemporal lobar degeneration or late-onset mental disorders, making a clinical diagnosis of the disease can be difficult (Wang et al., 2025). These illnesses may occasionally coexist with AD and have comparable clinical signs and symptoms (Beach et al., 2012; Hampel et al., 2021; Wang et al., 2022). In experimental laboratories, histochemical dyes (e.g., Congo Red, Thioflavin T, etc.) are frequently used for pathological examination to investigate whether amyloid proteins are present in tissues. Besides, mass spectrometry can be used to identify amyoid proteins and classify their type of accumulation and patterns (Vrana et al., 2009). Since the structures and sizes of proteins building up amyloid fibrils are highly varied (López de la Paz et al., 2002), more advanced techniques are required to detect amyloid’s mark and its type in particular tissues. Among known modern assays, liquid chromatography-tandem mass spectrometry (LC-MS/MS) is considered the most effective, stable, and accurate one. However, as a high-level technique with expensive chemicals used, applying it requires large budgets, skilled experimenters, and a longer time. To accelerate the screening process and save budgets, computational biologists have developed various in silico models using current computer-aided advances. Several studies have utilized computational methods to predict the propensity for $\beta$-amyloid formation (Orlando et al., 2019), determine the aggregation-prone areas (APRs) of amyloid proteins (Maurer-Stroh et al., 2010; Thangakani et al., 2014), and identify amyloidogenicity (Palato et al., 2019). Aggregation Nucleation Prediction in Peptides and Proteins (ANuPP), by Prabakaran et al. (2021), was developed using ensemble learning. This tool was designed to flexibly identify putative APRs in peptides and proteins, overcoming the shortcomings of many existing models at the time. With the continuous growth of empirical data on pharmaceutical, chemical, and biological sequences (Mendez et al., 2018), coupled with advancements in artificial intelligence (AI), there has been a growing focus over the past several decades on developing more efficient models for amyloid proteins, particularly leveraging bio-molecular representations in bio-cheminformatics (Nguyen-Vo et al., 2024; Chen et al., 2024). Advanced AI techniques have been utilized to analyze neuroimaging data and patient histories (Chen et al., 2017), aiding in the diagnosis and treatment of various neurological diseases (Pham et al., 2019). Attention mechanisms have been leveraged to improve model interpretability by focusing on the most relevant features in imaging data (Chen et al., 2021; Wu et al., 2024). Furthermore, hybrid models integrating recurrent neural networks (RNN) with attention mechanisms have been developed to capture temporal dynamics and essential features in patient data, resulting in enhanced prediction accuracy for AD-related outcomes (Li & Ma, 2022).

Related work

For years, many machine learning models have constructed to predict sequences using different representations, such as amino acid composition (AAC) (Bhasin & Raghava, 2004), composition transition distribution (CTDC) based on the percentage of particular amino acid property groups (Li et al., 2006; Dubchak et al., 1995; Chen et al., 2018), amphiphilic pseudo-amino acid composition (APAAC) (Chou, 2011), composition transition distribution difference (CTDD) based on distribution of amino acid properties in sequences (Chiti & Dobson, 2006; Vrana et al., 2009), and Dipeptide deviation from expected mean (DDE) (Saravanan & Gautham, 2015; Wang et al., 2020). RFAmyloid, developed by Niu et al. (2018), is one of the pioneered model designed to distinguish amyloid proteins from ordinary ones. It was created using random forest and different encoding schemes for sequence information extraction. They collected and curated protein sequence samples to obtain a dataset with 165 amyloid and 382 non-amyloid proteins for modeling. iAMY-SCM, by Charoenkwan et al. (2021), was developed the Scoring Card method to predict and analyze amyloid proteins. In their proposed method, a streamlined weighted-sum function was used in combination with thr propensity scores computed for dipeptides. iAMY-SCM was reported to achieve as good performance as RFAmyloid did based on metrics evaluated in cross-validation and independent testing (Charoenkwan et al., 2021). Charoenkwan et al. (2022) proposed AMYPred-FRL, an ensemble model with better performance reported. They combined six common machine learning algorithms, including logistic regression, k-nearest neighbors (k-NN), support vector machine (SVM), maximum gradient boosting (MGB), extremely randomized trees, and random forest (RF), and ten different sequence-based feature descriptors to construct 60 base models. In prediction, these base models returned 60 probabilistic features, which were then fed to the final meta-model (Charoenkwan et al., 2022). Most recently, Yang, Liu & Zhang (2023) proposed ECAmyloid, another ensemble machine learning model that detects amyloid proteins using diverse sequence-derived features. The sequence-based features combined information on the composition, evolution, and structure of the sequences. To select the most suitable models for ensemble learning, they used an enhanced classification selection method. The prediction outcomes of the meta-learner were based on the decision of all voters (base models) (Yang, Liu & Zhang, 2023).

Materials and Methods

Our proposed method

Bidirectional long short-term memory

Long short-term memory (LSTM), belonging to the family of RNNs, is designed to tackle the problem of gradient vanishing when the models are forced to learn long-distance sequential data (Hochreiter & Schmidhuber, 1997). To address this problem, gating mechanisms (forget, input, and output gates) and memory cells are employed to control flow of information. LSTM can maintain and update its cell state over long periods. LSTM’s design helps it effectively deal with many tasks from language processing to time series prediction. The mathematical expression of the Forget gate is as follows:

(1) $$f_t=\delta (W_{forget}\left[h_{t-1},X_t\right]+b_{forget}),$$where $\delta$ denote the sigmoid activation function, $X_t$ is the input vector at timestep $t$, and $h_{t-1}$ is the hidden vector from the previous timestep $t-1$. The weights $W_f$ and bias values $b_f$ are randomly initialized and gradually updated when the model learns. The mathematical expression of Input and Output gates are defined as:

(2) $$i_t=\delta (W_{input}\left[h_{t-1},X_t\right]+b_{input}),$$

(3) $$O_t=\delta (W_{output}\left[h_{t-1},X_t]+b_{output}\right).$$

The candidate memory cell ( $L_t$) contributes to adjusting the information in the current cell state ( $C_t$). The output $O_t$ and current cell state ( $C_t$) are used to create the current hidden stage ( $h_t$), as follows:

(4) $$L_t=\tanh (W_{cell}[h_{t-1},X_t]+b_{cell}),$$

(5) $$C_t=f_t\times C_{t-1}+i_t\times L_t,$$

(6) $$h_t=O_t\times \tanh \left(C_t\right).$$

Attention layer

Our attention layer accept hidden states $H=[h_1,h_2,h_3,\cdots ,h_n]$ returned from the LSTM layer. The vectors of hidden states $h_k$ are then nonlinearly transformed to create activated output $U=[u_1,u_2,u_3,\cdots ,u_n]$ using the Tanh function:

(7) $$u_k=\tanh ({\mathbf{W}}_k{h}_k+b_k),$$where $W_k$ denotes the weight matrices and $b_k$ refers to the offset quantity at timestep $k$. Certain operational factors during the shield tunneling process have a significant impact on the direction and location of the shield tunnel, thus they should be given additional consideration. The attention mechanism is applied to create the attention weight matrix ${\alpha }_k=[{\alpha }_1,{\alpha }_2,{\alpha }_3,{\alpha }_4\cdots ,{\alpha }_n]$. This matrix stores important information of all single intermediate states, described as:

(8) $${\alpha }_k=\frac{\exp \left(u_k^T{u}_{\mathrm{s}}\right)}{{\sum }_{k=1}^n\exp (u_k^T{u}_{\mathrm{s}})},$$where ${\alpha }_k$ refers to the normalized attention weight at timestep $k$ and $u_{\mathrm{s}}$ is the time series attention matrix which are randomly initialized. Finally, for each input sample, a computed weighted sum of hidden states is obtained to create an attention vector V for the next stage:

(9) $$V=\sum _k^n{\alpha }_k{h}_k.$$

Model architecture

The protein sequences, each represented as a distinct chain of letters, are encoded by the Embedding layer to form a sequence vector $X=(X_1,X_2,X_3,X_4,\dots ,X_t)$. First, the sequence vector length L is embedded after passing through the Embedding layer. The Embedding layer is defined by a vocabulary of 21 letters and an embedding dimension of 100. The embedding matrices, with dimensions $L\times 100$, are then input to the Bi-LSTM layer. After processing through the Bi-LSTM layer, the hidden dimension vectors at all timesteps are used to compute the attention scores using the Softmax function. The attention outputs, with dimensions $1\times 128$, are then passed through the first Fully Connected (FC) layer, normalized using 1-Dimensional Batch Normalization (BatchNorm1D) (Ioffe & Szegedy, 2015), and activated by the rectified linear unit (ReLU) function. Finally, the activated outputs are transferred to the second FC layer, where they are activated by the Sigmoid function to produce the final prediction outcomes. Figure 1 visualizes the architecture of the proposed model.

Results and discussion

Table 2 summarizes the performance of all implemented models, including model developed using our methods and those developed with other machine learning ones.

It can be observed that XGB is a promising model that can effectively learn features from protein sequences, as its metric values are remarkably greater than those evaluated for $k$-NN and SVM models. The AUROC values of the XGB model trained with the DDE and CTDD feature sets are 0.8819 and 0.8863, respectively. Additionally, the RF model trained with the DDE feature set has higher predictive power than models trained with the other feature sets, with an AUROC value of 0.8855. Compared to these machine learning models, our model achieves the highest performance with an AUROC value of 0.9126. The balanced accuracy and F1 score of our model also outperform the other models, with values of 0.8788 and 0.7447, respectively.

The comparative analysis between our model and the other computational frameworks is shown in Table 3. The results indicate that the model developed using our proposed method has better performance compared to the existing frameworks, with an AUROC value of 0.9126 and a balanced accuracy of 0.8788. Regarding the MCC, our model remains the best-performing model with a value of 0.7454. Our model’s F1 score is also greater than that of other models, with a value of 0.7447.

Model’s robustness examination

The performance of our model was evaluated over 10 random modeling trials, as summarized in Table 4. Across these trials, the model demonstrates robust performance, with an average AUROC value of 0.9167, indicating a high capability for distinguishing between classes. Similarly, the mean balanced accuracy of 0.8898 reflects a strong balance in sensitivity and specificity. The average MCC value of 0.7447 further confirms the reliability of the model in terms of binary classification correlation. Finally, the averaged F1 score of 0.7438 highlights its consistent precision and recall balance. These results demonstrate the stability and efficacy of the model, as evidenced by the low standard deviations across all metrics.

Limitations and future work

The proposed method, while demonstrating strong performance in amyloid protein identification using Bi-LSTM combined with an attention mechanism, does have certain drawbacks. One primary concern is the model’s reliance on sequence information alone. Although this approach captures essential patterns within protein sequences, it may overlook critical structural and functional information that could enhance predictive accuracy. For instance, complex folding processes often occur in amyloid proteins, and their 3D structures and interactions with other biomolecules closely influence their pathological effects. By focusing solely on sequence data, the model could miss these aspects, potentially limiting its generalizability to different types of amyloid proteins or other related neurodegenerative diseases.

Another limitation is the potential risk of overfitting, particularly given the complexity of the Bi-LSTM and attention mechanisms used. Overfitting occurs when a model learns to perform exceptionally well on the training data but fails to generalize to new, unseen data. If the training dataset is not sufficiently large or diverse, it exacerbates this risk. Moreover, the computational complexity of the model can pose practical challenges, especially when dealing with large-scale datasets or deploying it in real-time diagnostic settings. The interpretability of the model’s decisions is also a concern, as the intricate layers of Bi-LSTM and attention mechanisms can obscure the reasoning behind predictions, making it difficult for researchers or clinicians to trust and verify the model’s outputs.

For future work, it would be beneficial to explore the integration of additional types of data, such as structural information from 3D protein folding patterns or interaction data with other biomolecules. This multi-modal approach could provide a more comprehensive understanding of amyloid proteins and potentially improve the model’s accuracy and generalizability. Another promising direction is the development of techniques to enhance model interpretability, such as attention visualization or layer-wise relevance propagation, which would allow researchers to better understand which parts of the protein sequence contribute most to the predictions. Additionally, efforts should be made to reduce the model’s computational demands, possibly through model optimization or the use of more efficient architectures, to facilitate its deployment in practical, real-world scenarios. Finally, ongoing validation with larger and more diverse datasets will be crucial to ensuring the model’s robustness and applicability across different populations and conditions.

Conclusion

In this work, we presented an effective model for the in silico screening of amyloid proteins using attention-based Bi-LSTM. The introduction of an attention layer helps the model selectively focus on essential information and extract important features from amyloid protein sequences. Experimental results indicated that the model developed using our proposed method improved performance on the independent test set compared to other machine learning models as well as existing computational frameworks. Other metrics, including balanced accuracy, MCC, and F1 score, also demonstrated the robustness of the proposed method. This method can be modified to suit various dataset sizes and applied to address similar problems.

Supplemental Information