An integration of deep learning with feature embedding for protein–protein interaction prediction

Dataset

S.cerevisiae, Human and five species-specific protein–protein interaction datasets, which are collected from publicly available interacting protein databases (DIP) are used for performance evaluation. The S.cerevisiae core dataset (Guo et al., 2008) is chosen to confirm the parameters of the proposed model. The whole S.cerevisiae core dataset is made up of 11,188 protein pairs, including 5,594 positive protein pairs and 5,594 negative protein pairs. The human dataset (Huang et al., 2015) is also adopted to evaluate the effectiveness of the proposed method. It is collected from the Human Protein References Database (HPRD) and consists of 3,899 positive protein pairs and 4,262 negative protein pairs. To show the generality of model, five species-specific protein interaction datasets (C. elegans, E. coli, H. sapiens, M. musculus, and H. pylori) (Zhou, Gao & Zheng, 2011) are considered. These species-specific protein interaction datasets just contain positive samples. The positive interactions of each dataset are 4,013, 6,954, 1,412, 313 and 1,420.

The framework of DeepFE-PPI method

Representation learning can discover informative representations in a self-taught manner. Deep learning architecture can utilize multiple hierarchical layers to extract effective features. Leveraging the advantages of representation learning and deep learning, we proposed DeepFE-PPI framework, which consists of two main steps (Fig. 1): (1) residue representation learning; and (2) protein–protein interaction classification by deep learning.

Residue representation learning

Word2vec has been a very successful technique for representation learning in an unsupervised way. Word representations can reflect item meaning using the co-occurrence information of words in a corpus. The fixed dimensionality representations generated by Word2vec have been shown to be very useful for various NLP task. Except for processing words in text processing field, Word2vec can map other information units (for example, phrase, sentences or documents) to a low-dimensional implicit space. In this implicit space, the representation of each information element is a dense eigenvector. Supplemental Information (Section: Continuous Skip-gram Model) gives the details of Word2vec.

This motivates us to transform each residue into a fixed dimensionality eigenvector representation. In previous work, many common word definition and splitting approaches in bioinformatics are proposed (Dong, Wang & Lin, 2005; Tsubaki, Shimbo & Matsumoto, 2017). Tsubaki, Shimbo & Matsumoto (2017) adopted Word2vec to learn feature representations from a large protein database to address protein fold recognition problem. However, previous work all define a word as an n-gram item, and they all split the sequences into overlapping of the n-gram residues. 20 types of amino acids make up a protein sequence, the total number of possible n-grams is 20ⁿ. The larger n, the larger vocabulary size in the corpus. More vocabularies require more run time and memory consumption when training the Word2vec model. So, we set n as 1, which means that a residue is treated as a word and a protein sequence is treated a sentence. Compared to other protein representation methods, (1) this method can obtain different residue interactions from the original protein corpus. (2) the cost of run time and memory consumption is acceptable.

In the training process of word representation, a large corpus of sentences should be fed into the model to ensure sufficient contexts are observed. In order to derive the representation of different biophysical and biochemical properties that hidden in protein sequences, 558,590 protein sequences from Swiss-Prot database are adopted as the input of the Word2vec model. The protein sequences were downloaded in Nov. 2018. There are two available models in Word2vec: the continuous bag of word (CBOW) model and the Skip-gram (SG) model. According to previous studies (Smaili, Gao & Hoehndorf, 2018), it is found that the SG model has advantage on creating better quality vector representations. Therefore, this paper uses the Skip-gram model to derive residue vectors.

In this paper, Word2vec has been implemented using Gensim library. Hyper-parameters should be decided in the training stage. Hyper-parameters selection is a task dependent on choice and can vary from task to task. In the Skip-gram model, the target words are the words in the context, size denotes residue dimension, window denotes the maximum distance between the current and predicted word, min-count means the number of the ignored words with total frequency lower than a fixed value. Since the Swiss-Prot database just contains 25 different type residues, so the value of min-count in our experiment is set as 0. Parameters size and window are considered as the decisive parameters in the training of the Skip-gram model. After comparison of different parameter combinations, the value of size is set to be 20 and window is set as 4. The specific Word2vec parameters selection process is shown in Section Parameter selection of Results. Figure 2 shows the general process of residue representation. Protein sequences from Swiss-Prot database are adopted to train the Word2vec model. The output of the Word2vec model are residue eigenvectors. Each protein sequence can be represented as a concatenation eigenvector of residue vectors.

Deep learning

Deep learning supports highly flexible architectures, such as DBNs (Deep Belief Networks), CNNs (Convolutional Neural Networks), RNNs (Recurrent Neural Networks) and so on. Different deep learning architectures are suitable for different specific applications. For example, DBNs are good at analyzing internal correlations in high-dimensional data, CNNs are capable of capturing complex features, and RNNs are suitable for the analysis of sequential information (Goodfellow, Bengio & Courville, 2016). Considering residues’ interactive impacts on each other and the possible high dimension of a protein sequence, we select DNN as the most appropriate or ‘best fit’ deep learning architecture to predict protein–protein interactions. Supplemental Information (Section: Deep neural network) gives the details of DNN.

DNN is a very successful machine learning technique for classification in biology, but its input should be vectors with fixed dimension. This is a dreadful limitation because proteins are intrinsically expressed as sequences of arbitrary length. A simple measure is to uniform the length to a fixed size m. As a result, every protein sequence is represented as a vector of size D∗m (D is residue dimension and m is the fixed number of residues). If the length of a protein sequence is shorter than m, zero is appended to the short eigenvector to ensure every protein sequence has the same shape. These two embedding vectors are taken as input data of deep learning model. Our deep learning framework is composed of two separate DNN modules to extract high-level features that hidden in original embedding vectors and a joint module to integrate both DNN modules outputs to classify the input protein pair as interaction or non-interaction. The deep learning flowchart is shown in Fig. 3. For both DNN modules, we configure the same number of layers and hidden units. The number of fully connected layers is set as 4 and the corresponding units in each layer is 2,048, 2,014, 512 and 128, respectively. The main role of DNN module is to extract high-level feature, eliminate noise and reduce data dimension. The joint module takes the concatenated last hidden vectors of the both DNN modules as input via two fully connected hidden layers. The output layer utilizes a softmax function to predict interaction probability. In DeepFE-PPI, a batch-normalization layer and a dropout layer are added to each fully connected layer except the output layer. Rectified Linear Unit (ReLU) function is chosen for all layers except the final layer. During training, the models are optimized using Stochastic Gradient Descent (SGD) algorithm. L2-regularization term is added to the loss function as well, by adding the sum of the squares of all weights in the network.

Generally speaking, we first use unsupervised representation learning method Res2vec to generate residue representations. Then each protein sequence is uniformed into a vector representation with the same length. After that, separate DNN is established for feature extracting, the merged layer is for feature integration and the last layer is for classification.

Parameter selection

We perform the protein–protein interaction prediction by the derived Res2vec model for residue vectors generation and deep learning technology for classification, therefore, there have two parameter sets needed to be confirmed. Previous literature (Li et al., 2018) has shown that training corpus size, residue vector dimension, and the algorithm adopted in Word2vec are the main factors that affect the quality of residue representations and the deep learning classification results. In addition, the hyperparameters of the deep learning model also determine prediction results. In order to get the optimal residue eigenvectors and deep learning configurations, we train the Skip-gram model on Swiss-Prot database which includes 558,590 proteins sequences and explore the effect on protein–protein interaction prediction over different parameter combinations on the S. Cerevisiae Core dataset. As the combinations of parameters increase exponentially and trying all of them is impossible in practice, we choose the most important hyper-parameters such as residue dimension, window size, network depth and protein length. Table 1 summarizes the most important hyper-parameters and their value intervals.

In the Skip-gram model, the two parameters of window size and residue dimension are most important. As the propensity for protein interactions is not a local property of single amino acid positions, it is expected that using the characteristics and patterns of neighboring residues could be helpful in the prediction of protein interactions. Previous studies (Murakami & Mizuguchi, 2010; Dhole et al., 2014) analyzed the entropy differences for various window lengths and finally emphasized that a nine-residue window size would be optimal for protein–protein interaction prediction problems, the initial length of the Skip-gram windows is taken as 4. Considering the run time and memory consumption, we train the residue vectors in case of window size [4, 8, 16, 32]. Residue dimension has a high impact on the classification, too. Because of our machine’s memory (16G) and residue dimension determines the size of a protein vector, residue dimensions are set as [4, 8, 12, 16, 20]. Other hyper parameters are set to default values in the toolkits of Word2vec. Table 2 shows the predictive performance of DeepFE-PPI on the S. Cerevisiae Core dataset with different combinations of residue dimension and window size. The experiment results show that when residue dimension is set as 20 and window size equals to 4, the model achieves the best performance with mean Accuracy 0.9461, mean Precision 0.9580, mean Recall 0.9333 and mean MCC 0.8927. The other measurements of Specificity, F1, AUC-ROC and AUC-PR are 0.9589, 0.9454, 0.9837 and 0.9868, respectively.

For deep learning, prediction results are determined by input data and model complexity. The deeper the DNN network is, the more abstract the features learned and the more accurate the model achieves. The selection of the network depth is related to the specific application and experimental data, so the appropriate network depth should be determined by experimental verification.

In the experiment, protein length is also an important factor that could influence the final result. We have experimented four different values of protein length starting from the average length with n step size as 100. The range of protein length is [n, n + 100, n + 200, n + 300]. So we discuss four different network structure depths in combination with different network inputs. The networks are incremented by one layer of fully connected layers. Tables 3, 4, 5 and 6 show that model performance increases first and then decreases when network depth increases from 2 to 5 with the condition that the protein length is set as 550, 650, 750 and 850. If network depth is not enough, the model is under-fitting and cannot capture deep features. While a network that is too deep will increase model complexity and training time. Due to protein length and the limitations of an experimental platform, DeepFE-PPI achieves the best performance when protein length is set as 850 and the number of layers in the network is 4.

Hyper parameters are often selected in a statistically rigorous way via validation or cross-validation examples that are held out from training. We used five-fold cross-validation (training dataset proportion: 72%, validation proportion: 8%, testing proportion: 20%) and trained a neural network with four hidden layers on the S. Cerevisiae Core dataset. Usually, the training set is used to train models with different hyper-parameters, which are then assessed on the validation set. The best model configuration is then assessed by comparing the often used statistical measures, such as accuracy, recall, specificity, precision and MCC, on the test set.

Performance comparisons with state-of-the-art algorithms

In order to evaluate the prediction performance of DeepFE-PPI, we compared the results of the proposed method with six state-of-the-art algorithms (Support Vector Machine (SVM), Random Forest (RF), Decision Trees (DT), Native Bayesian (NB), (K-Nearest Neighbor, KNN) and Logistic Regression (LR)). The experiments are carried on the S. Cerevisiae Core dataset using five-fold cross validation procedure. From methods in Li, Zhang & Liu (2017) and Maeda (2018), different machine learning models have very different algorithmic principles and parameters, only when comparing the algorithms with the most optimized parameters would make sense. Furthermore, training accuracy as well as training time should be compared as the evaluation criteria. For each algorithm, we fine-tune the model parameters by using grid-search to obtain the optimal parameters. Note, Naive Bayes algorithm model parameters via ‘partial-fit’ method and the sole parameter ‘priors’ uses default value that denotes the prior probabilities of the classes, so we have no training parameters for setting. Due to the fact that SVM performs slower than the other five algorithms, it takes about 20 h once time based on the configuration of our computer (a PC with an Intel Core i5-7400 processor and a 16G memory running windows 10), so the parameters are set to default values.The specific parameter details and the final recommended settings of each algorithm are described in Table 7.

The proposed deep learning method and the other six different classification results are listed in Fig. 4. Among six state-of-the-art methods, Random Forest (RF) achieves 93.96% accuracy, 96.96% precision, 90.76% recall, 97.16% specificity, 88.10% MCC, 93.76% F1, 96.93% AUC-ROC and 97.69% AUC-PR. The classification results of DeepFE-PPI are 0.82% accuracy, −0.51% precision, 2.23% recall, −0.59% specificity, 1.52% MCC, 0.93% F1, 1.61% AUC-ROC and 1.14% AUC-PR higher than RF. Experimental results show that DeepFE-PPI performs better in detecting protein–protein interactions in terms of accuracy, recall, MCC, F1, AUC-ROC and AUC-PR.

From Fig. 4, we can see that the result of RF is also excellent. However, this method requires manual selection of features. Firstly, it takes a lot of time to find suitable features. If the features are not found properly, the results will be affected. Second, feature selection and data training are a process of separation and cannot assist each other. The main advantage of the deep learning is the automation of feature selection. It can automatically process the features that can express the essence of things according to the follow-up training process, so as to realize the automation process without manual intervention. Furthermore, RF is sensitive to the noise in the original input, frequent noise brought by features construction from the low-level data may make the subsequent classifier to learn wrong knowledge. While a deep learning framework consists of neural networks stacked together (Hinton & Salakhutdinov, 2006), the stacked layers help to reduce the noise effects in the original input. Due to limitations in our computer hardware configuration, the deep learning model we constructed is not strong enough, we believe that deep learning model can perform better than RF in all indexes if the hardware configuration is better.

Table 8 shows training time with the most optimized parameters between different algorithms. As seen from Table 8, the time of DeepFe-PPI ranks fourth in all seven methods. Naive Bayes training time is the shortest, but its results are the worst.

Performance comparisons with existing methods on the S. Cerevisiae Core Subset

Our analysis first proceeds on the S. Cerevisiae Core dataset. The results of different methods with five-fold cross validation on the S. Cerevisiae Core dataset are shown in Table 9. From Table 9, we can see that DeepFE-PPI achieves good prediction performance with a 94.78% average accuracy, 92.99% average recall, 96.45% average precision and 89.62% average MCC. Compared with our previous proposed deep learning-based algorithm DeepPPI, DeepFE-PPI boosts about 0.35% accuracy, 0.93% recall, −0.20% precision and 0.65% MCC. However, DeepPPI used five different category features to represent a sequence of a protein. The feature extraction process is more complicated than Res2vec, which directly represent a residue. Both on classification results and protein representation step, DeepFE-PPI performs better than Deep-PPI. From the overall perspective, DeepFE-PPI outperforms better than most other approaches (10 of the 13 methods with the whole measure metrics). But it seems that (Wang et al., 2017b) obtained the highest performance according to each evaluation metric and the following one is RFEC (Song et al., 2018). Note that, the Legendre moments (LMs) (Wang et al., 2017b), Zernike moments (ZM) descriptor (Wang et al., 2017a) and the evolutionary information (Song et al., 2018) all extracted discriminatory information embedded in the position-specific scoring matrix (PSSM) which is generated by Position-Specific Iterated Basic Local Alignment Search Tool (PSI-BLAST) (Altschul et al., 1997). Above three methods run much slower because it is required to run BLAST against a huge protein NR database to generate a PSSM matrix as its feature. For Huang et al. (2015), the performance on the S. Cerevisiae Core dataset is really better than DeepFE-PPI while the performance on the human and five independent datasets are worse than DeepFE-PPI. Especially that using the S. Cerevisiae Core dataset as training set and five independent across species datasets as test sets, the accuracies of the approach Huang et al. (2015) proposed are 33.92% (E.coli), 18.81% (C. elegans), 17.78% (H. sapiens), 17.82% (H. pylori), and 20.13% (M.musculus) lower than DeepFE-PPI, which indicates that the generalizability of Huang’s model (Huang et al., 2015) is quite weak. More details of independent datasets are shown in Table 9. Considering computational time, feature extraction process and model generalizability, DeepFE-PPI is more appropriate for protein–protein interaction prediction.

Performance comparisons with existing methods on the human dataset

Consequently, human dataset is employed as a test dataset. The average prediction results with five-fold cross-validation over twelve different approaches are given in Table 10. Table 10 clearly shows that DeepFE-PPI achieves the highest performance in terms of accuracy (98.71%), recall (98.54%) and MCC (97.43%). Compared to our previous work (Du et al., 2017), the value of accuracy, recall and MCC has improved 0.57%, 1.59% and 1.14%, respectively. The second-best value (98.77%) of precision is just slightly lower (−0.36%) than our previous work (99.13%). Since our previous work has achieved best in predicting protein–protein interactions, so taken together, this comparative analysis demonstrates that residue representations detected by Res2vec achieve comparable classification performance, in contrast to sophisticated feature extraction step.

Performance comparisons on independent datasets

To further validate the performance of our proposed method, we also compare the predictive performance of our method on five other independent species datasets. In this experiment, the selected 11,188 samples of S. cerevisiae core subset are used as training data, five other species datasets (E. coli, C. elegans, H. sapiens,H. pylori and M.musculus) are used to assess the performance of DeepFE-PPI. Table 10 gives the detailed performance comparisons among different methods. The value of accuracies are 100.00% (E. coli), 100.00% (C. elegans), 100.00% (H. sapiens), 100.00% (H. pylori), and 100.00% (M.musculus) by DeepFE-PPI, respectively. From Table 11, we can see that DeepFE-PPI is superior to DeepPPI on all five across species independent datasets. Compared with the other methods, our method obtains the highest accuracy on five independent species datasets. The promising results demonstrate that the proposed method has higher accuracy and better generalizability.

Park and Marcotte’s evaluation scheme

Typical cross-validation for pair-input methods divide available data into a training set and a test set, ignoring that the way of test datasets constructed can significantly impact the performance of the pair-input predicting programs (Park & Marcotte, 2012). Based on the component-level overlap of test datasets, test dataset can be partitioned into three distinct classes: C1, test pairs sharing both proteins with the training set; C2, test pairs sharing only one protein with the training set; C3, test pairs sharing neither protein with the training set. We design similar testing datasets according to Park and Marcotte’s procedure (Park & Marcotte, 2012). S. Cerevisiae Core dataset has 2,530 different protein sequences. We randomly split the whole protein sequences into two parts: part A has protein sequences (numbers) while the rest serve as part B. Then we pick out two interacting protein sequences sharing both proteins in the 2,230 protein sequence as training and C1 type datasets (4/5 as training dataset and 1/5 as C1 type dataset). Protein pairs that one protein sequence in part A and the interacting protein sequence in part B is considered as C2 type dataset. Protein pairs that both from part B constitute C3 type dataset. The performance on each dataset is showed as follows (Table 12):

Visualization of residue representations

Due to the input sequence representation determines the regulatory properties of each protein. Hence, it is critical to determine the optimal dimension of each residue for protein–protein interaction prediction. Maintaining a consistent data split ratio with five-fold cross validation, 72% of the dataset is split as a training set, 8% of the dataset is split as a validation set and 20% for model testing. We use DeepFE-PPI model to predict interactions on the S. Cerevisiae Core dataset and obtain 6 performance results under 6 residue eigenvector sets. Each result corresponds to a special residue dimension (from 4 to 24 with 4 steps). The performance with different residue dimensions is shown in Fig. 5. From Fig. 5, we can see that when the residue’s dimension increases from 4 to 20, the whole performance measure metrics (accuracy, recall, precision and MCC) is gradually rising. The classification results decrease when the residue dimension continues to increase four steps starting from 20. DeepFE-PPI achieves superior performance with the dimension of residue equals to 20. Considering run time and memory size, the optimal residue dimension in this article is selected as 20.

To visualize the effect of residue dimension for residue quality, we applied principal component analysis on the optimal dimension residues. The top five ratios of the variance of each PCs are 0.6421, 0.1412, 0.0649, 0.0347, 0.0279. The first two principal components are adopted to cluster 25 residues. As can be seen from Fig. 6, the common 20 residues are clustered together while the other five uncommon residues are far apart, which also explains the rationality of Res2vec.In addition, we also show the memory and run time (please see Supplemental Information: Memory and Time).