Protein function prediction with gene ontology: from traditional to deep learning models

Supervised learning

Supervised learning is an essential approach for in silico protein function prediction. Supervised models learn to annotate function guided by a training dataset, which is a group of characterized proteins with reliable, experimentally confirmed annotations. After the learning stage, a model that captures the relationship between feature and function is produced, and used to predict the GO terms for novel amino acid sequences.

Deep neural network (DNN) is a feed-forward architecture. It generally comprises an input layer, multiple hidden layers, and an output layer. Input data are processed in a unidirectional manner along the layers, from the first to the final stage. With respect to GO-based functional annotation, DNNs have been employed as multi-task DNNs (MTDNNs) (Fa et al., 2018) or a series of MTDNNs (Rifaioglu et al., 2019).

Convolutional neural network (CNN) was initially developed to handle two-dimensional images for handwritten digit recognition (LeCun et al., 1990). However, it has since become an effective architecture not only for multi-dimensional data, but also for one-dimensional input, such as sentences or genomic sequences. A CNN model starts with convolutional layers, whose units are feature maps. Each unit is obtained by computing convolution operations between local patches of a unit in the preceding layer and a filter (set of weights) (LeCun, Bengio & Hinton, 2015). Non-linearity pooling layers are stacked to convolutional layers, enabling the integration of different-scale features in this architecture. CNNs can be applied in AFP either alone (Kulmanov, Khan & Hoehndorf, 2018; Cai, Wang & Deng, 2020; Du et al., 2020) or in combination with other architectures (Zhang et al., 2020; Spalević et al., 2020).

Recurrent neural network (RNN) is a deep learning architecture developed especially for sequential data. RNNs have a DNN backbone, and the units of the hidden layer are connected. Therefore, a hidden unit receives information from the input layer at the current time step, and also from the hidden units at the preceding stage (LeCun, Bengio & Hinton, 2015). Long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) is a variant of the RNN model, developed to capture long-term dependencies between input sequences. Bi-directional LSTM (Bi-LSTM) is an LSTM model that processes data in two directions, forwards and backwards. These neural networks are employed to process protein sequences as a natural language (Cao et al., 2017) or are combined with the CNN model (Zhang et al., 2020) to provide GO annotations.

Another type of neural network, fully connected deep network (FCDN), is a series of fully connected layers. This model was utilized for converting input vectors from InterPro and predicting GO terms (Zhang et al., 2019).

Unsupervised learning

Unlike supervised learning, unsupervised learning independently mines hidden patterns in input distribution. It is crucial for exploring unlabeled data and provides important information for supervised architectures. Typically, unsupervised learning models are employed for clustering, reducing dimensions, and transforming data.

Autoencoder (AE) (Baldi, 2012) is an unsupervised learning model developed for assigning GO terms to amino acid sequences. AE is a neural network for data transformation. After encoding and decoding, the target output size is the same as that of the input. Using a trained AE model, GO predictions for proteins can be deduced directly from the generated matrix (Chicco, Sadowski & Baldi, 2014). Alternatively, low-dimensional feature representations in the hidden layer are extracted and fed into an SVM classifier for final classification (Gligorijević, Barot & Bonneau, 2018) or a CNN model (Peng et al., 2020) for final classification.

Restricted Boltzmann machine (RBM) (Salakhutdinov & Hinton, 2009) has one hidden layer for representing latent features and an input layer encoding the observed data. For GO annotation, units of the input layer are the available GO terms (Zou, Wang & Yu, 2017). Zou, Wang & Yu (2017) introduced a deep RBM (DRBM), where multi-layer RBMs are trained and unfolded, resulting in protein function prediction.

Finally, generative adversarial network (GAN) (Goodfellow et al., 2014) comprises two networks, i.e., a generative model that produces synthetic realistic data and a discriminative model that evaluates whether the data are real or not. In AFP, the GAN model improves the prediction by creating synthetic features for supervised classifiers (Wan & Jones, 2020), or by exploiting ontology correlations (Seyyedsalehi et al., 2021).

Sequence-based models

One of the earliest deep learning-based methods for GO annotation predictions is the one proposed by Chicco, Sadowski & Baldi (2014). The authors compared two annotation solutions, i.e., tSVD, presented in Pinoli, Chicco & Masseroli (2013), and the AE neural network. The output form of the two methods was similar; however, the latter performed better on six different datasets. Later, instead of using the AE architecture, (Zou, Wang & Yu, 2017) proposed DRBM for annotating the gene products of four model species, Homo sapiens, S.cerevisiae, Musmusculus, and Drosophila.

ProLanGo (Cao et al., 2017) is the first tool that applies Neural Machine Translation (NMT) developed by Google in AFP. The authors converted amino acid sequences and GO terms into “ProLan” and “GOLan” languages, respectively. The GO annotation of a new protein was then generated after being translated using a model with three RNN layers. In another study, artificial neural networks proposed by Szalkai & Grolmusz (2018a) utilized six CNN layers of increasing depth. Based on that work, SECLEF (Szalkai & Grolmusz, 2018b) has been designed to train and test the GO function and UniProt protein family using biological sequences as input. The user can implement their self-configured models in a downloadable program or run two pre-trained models (Szalkai & Grolmusz, 2018a) on a web server. DeepSeq (Nauman et al., 2019) is another method for assigning GO terms to amino acid sequences, based on convolutional layers. However, the authors only predicted the five most frequent MF ontologies for H. sapiens proteins.

Using the DNN architecture, DEEPred (Rifaioglu et al., 2019) implements a stack of multi-task feed-forward DNNs. Each individual network is built for a specific GO term level on DAG, which allows the hierarchical post-processing of predictions. Protein sequences are experimentally represented by three types of descriptors(subsequence profile map, pseudo-amino acid composition, and conjoint triad feature), with subsequence profile map performing best in the analysis.

DeepGOPlus (Kulmanov & Hoehndorf, 2020) has been developed to overcome the existing limitations of DeepGO (Kulmanov, Khan & Hoehndorf, 2018), such as sequence length, unavailable PPI features, and the number of GO labels. Functional annotation is predicted by a multi-layer CNN structure in combination with sequence similarity. Lately, TALE (Cao & Shen, 2021) has been developed to generate GO predictions by integrating sequence patterns based on transformer encoder and the joint similarity of sequence-term. Similar to DeepGOPlus, TALE is also combined with sequence similarity as TALE+ model to enhance its performance. Meanwhile, PFP-WGAN (Seyyedsalehi et al., 2021) is one of two latest ideas that use GAN to infer the functionalities of proteins. While the generator network processes raw sequences, the discriminator takes two inputs; one consists of annotated proteins from the SwissProt database and the other of raw sequences with respective annotations synthesized from the generator.

Integrated data-based/Structure based-models

The functional assignment of amino acid sequences from 3D structures was proposed by (Tavanaei et al., 2016). The main analysis involved a CNN model that was trained and tested on five datasets of human proteins, each of which had been assigned two GO terms. However, the prediction has not been propagated on the DAG tree for inherited GO terms.

DeepGO (Kulmanov, Khan & Hoehndorf, 2018) is a functional annotation server based on CNN architecture that had been devised to learn features from amino acid sequences and the PPI network. GO labels are assigned to input proteins through a hierarchical classification structured as a DAG tree. DeepAdd (Du et al., 2020) was inspired by DeepGO server and provides a solution for AFP, utilizing a CNN framework to learn vector representations from sequences and additional information. However, sequences are processed into k-mer embedding by Word2Vec, instead of the tri-gram embedding suggested in DeepGO. Further, in DeepAdd, the protein sequence profile (SSP) is added if the PPI network information is not available. Based on a similar concept of combining primary protein structure and PPI, GONET (Li et al., 2020), a novel model, was built by employing CNN, RNN, and Attention layer for human and mouse sequences.

Working on a dataset identical to the one collated by FFPred3, Fa et al. (2018) tested an MTDNN that treated protein function prediction as a multi-label classification problem. Here, the solution consisted of layers shared by all tasks (GO labels), which are stacked in parallel with task-specific layers. DeepFunc (Zhang et al., 2019) is a novel predictor that surpasses DeepGO, FFPred3, and BLAST. First, protein domain, family, and motif information is queried from InterPro and encoded before passing through fully connected layers. Next, topological features of PPI are obtained by the Deepwalk algorithm. Finally, the method concatenates two types of features (sequence- and network-based input) to fit an FCDN. The architecture of DeepGOA (Zhang et al., 2020) is more sophisticated than that of DeepFunc. In addition to information generated by InterPro and PPI, global and local semantic features of amino acid sequences are extracted by Bi-LSTM and a convolutional layer, respectively. Using the same types of features, SDN2GO (Cai, Wang & Deng, 2020) utilizes three sub-models for each information source, with all outputs integrated in the final weighted model.

In another study, deepNF (Gligorijević, Barot & Bonneau, 2018) was constructed by multimodal deep AE to capture hidden information in proteins from different types of interaction networks. In that suggestion, feature representations are extracted to train the final SVM classifier to produce GO terms for human proteins. DeepMNE-CNN (Peng et al., 2020) has a superior performance than deepNF in the human data by utilizing CNN layers instead of SVM for the classification model. On the other hand, FFPred-GAN (Wan & Jones, 2020) utilizes GAN for feature enrichment to feed conventional machine learning models, especially SVM. Real features are biophysical information extracted from raw amino acid sequences by FFPred, and the generator takes latent variables to augment the synthetic samples.

Data

We used the finalized benchmark set provided in the CAFA3 report (Zhou et al., 2019). The statistics are presented in Table 3. There are two sequence types in the benchmark dataset, NK (no-knowledge) and LK (limited-knowledge), which have been coined in CAFA2 (Jiang et al., 2016). Based on the timeline of the challenge, t₀ is the deadline for the prediction submission of the target sequences. Then, experimental annotations for a subset of target sequences are accumulated until t₁, to complete a benchmark dataset for the performance evaluation. The NK proteins are those that had not had any experimental annotations in three GO domains at t₀, but had at least one GO term with experimental evidence code between t₀ and t₁. The LK sequences are those that had been experimentally annotated in one or two GO domains at t₀, and gained at least one experimental annotation in the other GO domains between t₀ and t₁.

Methods

The servers of several tools have been updated recently, but we need to control their training data or databases used, in which do not include sequences in the CAFA3 benchmark. Therefore, we chose INGA, which was one of the top models in CAFA3, as a representative conventional tool. DeepGOPlus is a recently developed representative annotation system that is based on deep learning. In addition to these two tools, other studies that performed their analysis on the CAFA3 dataset are also summarized in this part.

For DeepGOPlus, we re-trained the model with the CAFA3 training sequences and generated predictions for the CAFA3 benchmark set based on their source code provided. For INGA, we obtained predictions for the CAFA3 benchmark based on the predictions of CAFA3 targets, which were provided on their website.

Model performances were evaluated based on protein-centric metrics (F_max, precision, and recall) provided by the CAFA3 assessment tool. The area under the precision–recall curve (AUPR), which is a measurement of highly imbalanced data, was also considered. Results regarding the performance of other methods, including Naive, BLAST, FFPred3 and DEEPred, were derived from DEEPred paper (Rifaioglu et al., 2019).

In terms of evaluation, two modes were designed. In the partial mode, the scores of each method were computed on the sequences that have at least one GO term predicted at any threshold. Conversely, scores in the full mode were computed on all protein sequences in the benchmark set.