Improving drug–target affinity prediction by adaptive self-supervised learning

Data

The proposed model was first evaluated on five benchmark datasets of DTAP, namely, the Kiba (Tang et al., 2014), Davis (Davis et al., 2011), DTC (Tang et al., 2018), Metz (Metz et al., 2011), and Tox-Cast (US Environmental Protection Agency, 2015). The simple statistics for the sample information of these datasets are given in Table 1. It can be seen from Table 1 that there are only 2,111, 68, 5,983, 1,471, and 7,657 drugs and only 229, 442, 118, 170, and 328 targets on the above datasets. As a result, the prediction model could be hardly well trained only by these samples.

The proposed model was also evaluated on three benchmark binary classification datasets of compound–protein interaction (CPI) prediction, namely, the BindingDB (Gao et al., 2018), C.elegans (Liu et al., 2015), and Human (Liu et al., 2015) datasets. The simple statistics for the sample information of these datasets are given in Table 2. It can be seen from Table 2 that there are only 852, 1,767, and 1,696 drugs, and only 1,052 and 1,876 targets on the Human and C.elegans datasets. Although there are 53,253 targets on BindingDB, the used positive and negative samples are only 39,747 and 31,218. As a result, the problem of sample shortage still exists in CPI.

Problems of SSL usd in DTAP and our solution strategy

SSL can utilize a large number of unlabeled samples to train the model, which can be used to overcome the problem of sample shortage, where RoBERTa is a typical SSL framework (Liu et al., 2019). RoBERTa was first used for text processing and got good results, which is trained with dynamic masking and removed the next sentence predicts loss (Liu et al., 2019). Because the simplified molecular input line entry system (SMILES) of the drug and the sequence of the target are similar to the sentence of the text, RoBERTa is considered to extract features for DTAP. However, RoBERTa cannot be directly used in DTAP, for the following reasons:

Firstly, the input of self-supervised learning is very different with the input of DTA. Self-supervised learning processes drug and target samples independently, while DTA prediction requires the integration of drug-target pairs. Obviously, this gap makes that the self-supervised learning hardly learns useful features for DTA prediction.

Secondly, the input is very different between text processing and DTAP. One difference is that the lengths of the SMILES and sequence are much longer than that of the sentence of the text. Simple statistics of lengths of SMILES and sequences are presented in Figs. 1 and 2, where the rectangle represents the interquartile range, top, median, and bottom lines represent the upper limit, median, and lower limit of the lengths. It can be seen from Fig. 1 that the median lengths of SMILESs are respectively about 46, 53, 48, 45, and 28. It can be seen from Fig. 2 that the median lengths of sequences are respectively about 620, 632, 673, 631, and 479. Each word could be represented by a long vector in RoBERTa, which makes that the dimension of the hidden state for a long sequence could be very high. Another problem is that the repetition probability of words in SMILES and sequence are much larger than that of the sentence, which makes the positional information very important in training the RoBERTa, and then outputs of similar sequences with different positions could be very different. As a result, the data distribution of DTAP is sparser.

Thirdly, the output of the RoBERTa could be not meeting the needs of DTAP. The positional feature for a sentence could be important in text processing, but the positional feature for a SMILES or a sequence could be not very important in DTAP, as the DTAP is mainly determined by core fragments of the SMILES and sequence (Lin, Zhang & Xu, 2020; Jin et al., 2023), which is the principle of fragment-based drug design (Shi et al., 2020). An example can be shown by Fig. 3, where the core fragment of the drug is marked by a rectangle. It can be seen from Fig. 3 that the core fragment of the drug and the two core fragments of the target mainly determine the DTA. As a result, only the inner positional feature of the core fragment is important. Specifically, the positional feature between the core fragment and other fragments is bad for DTAP. Furthermore, if the pooling result of RoBERTa is used for DTAP, features of the core fragment could be overwhelmed. And if the whole result of RoBERTa is used, the dimension of features could be too high.

Fourthly, the training goal of the RoBERTa is very different from that of the DTAP. The training goal of the RoBERTa is to fill the dynamic masking. But the training goal of the DTAP is minimizing the mean square error between the prediction score and the actual score. Furthermore, because labeled training samples are too limited, the pre-trained RoBERTa is hardly fine-tuned by these labeled training samples. As a result, features extracted by RoBERTa should be further processed by a supervised method and cannot be directly used for DTAP. Specifically, features extracted by RoBERTa should own enough ability to describe the input and cannot lose much information.

To overcome the input problem, RoBERTa should be trained on fragments that are much shorter than the raw SMILES and sequence. To overcome the output problem, the output should be features of fragments and their relationship. To overcome the training goal problem, RoBERTa should mainly extract low-level features, which should own enough ability to describe the input and cannot lose much information.

Based on the above analysis, an ASSL based on RoBERTa is designed to extract low-level features. In ASSL, RoBERTa is trained on a large number of unlabeled fragments, then the feature is extracted by RoBERTa on the adaptive sliding window. As a result, the extracted feature has enough ability to reconstruct the fragments and to describe the relationship among the fragments. Furthermore, the features extracted by ASSL are further processed by a 2D CNN to extract high-level features, as the ASSL is only used to extract low-level features, where the 2D CNN can highlight the useful fragments and their neighbor relationships, and reduce the negative positional features among fragments.

ASSLDTA structure

The structure of ASSLDTA is shown in Fig. 4. It can be seen from Fig. 4 that ASSLDTA consists of five parts, such as ASSL, 2D CNN, GCN, and fully connected neural network (FCNN). In ASSL, low-level features of drugs and targets are extracted, which can be used to reconstruct fragments and their neighbor relationships. In the 2D CNN, the high-level features of drugs and targets are further extracted, which can highlight useful fragments and their neighbor relations. In GCN, graph-based high-level features of drugs are extracted since drug molecules have a typical graph structure. In FCNN, the high-level features extracted by two 2D CNNs and one GCN are integrated to predict the DTA. Here, it’s worth noting that ASSL specifically focuses on extracting low-level features rather than directly tackling high-level ones, due to four identified issues with the current application of self-supervised learning (SSL) directly to DTA prediction.

Features extracted by ASSL together with 2D CNN

ASSL consists of ASSL training and ASSL-based feature extraction. The ASSL training is first introduced. According to the motivation in “Problems of SSL usd in DTAP and Our Solution Strategy”, the data prepared for RoBERTa should be the fragments of the drug or the target, and the lengths of fragments cannot be too large. However, it is still difficult to automatically divide appropriate fragments for the drug and target. Specifically, the lengths of automatically divided fragments may be very different, which also adversely affects the training of the model. As a result, fixed-length fragments of the SMILES and sequence are prepared for the RoBERTa. We opted for RoBERTa over BERT due to our implementation of a two-phase feature extraction approach. In the pre-training phase, our objective is to optimally preserve the intrinsic characteristics of individual fragment. Conversely, the interaction and relationship among these fragments, which constitute the inter-fragment features, are intended to be developed and refined during the subsequent supervised training phase.

Given a training sequence $d_i=[d_{i,1},d_{i,2},\dots ,d_{i,pi}]$ whose length is p_i, the length of the fragment $l$, the k-th fragment of $d_i$ can be defined as:

(1) $$s_i^k=[d_{i,k\ast l},d_{i,k\ast l+1},\dots ,d_{i,k\ast l+l-1}]$$

It can be seen from Eq. (1) that $d_i$ is divided into $p_i/l$ non-overlapping fragments. The adjacent fragments do not overlap, for the reason that the same word would be masked in one fragment but not masked in neighbor fragments when adjacent fragments overlap with each other.

After processing all training drugs by Eq. (1), the RoBERTa model ${\stackrel{⌢}{f}}_T$ of the drug can be trained by hugging face toolbox (Transformer, https://github.com/huggingface). The CHEMBL (Gaulton et al., 2017) dataset is used for training, which contains 2,105,464 drugs. Similarly, after processing all training targets by Eq. (1), the RoBERTa model ${\stackrel{⌣}{f}}_T$ of the target also can be trained by the Hugging Face toolbox (Transformer, https://github.com/huggingface). The Swiss-Port (Swiss-Port dataset, http://www.gpmaw.com/html/swiss-prot.html) dataset is used for training, which contains 565,928 targets. Because ${\stackrel{⌢}{f}}_T$ and ${\stackrel{⌣}{f}}_T$ are trained on the fragments, they have enough power to extract the features for the fragment.

After training the ${\stackrel{⌢}{f}}_T$ and ${\stackrel{⌣}{f}}_T$, they can be used to extract the low-level features for the drug and target. The processing is named as ASSL based feature extraction. In the feature extraction, fixed-length fragments of the SMILES and sequence also should be prepared for the ${\stackrel{⌢}{f}}_T$ and ${\stackrel{⌣}{f}}_T$.

Given a testing sequence $di=[d_{i,1},d_{i,2},\dots ,d_{i,pi}]$ whose length is p_i, the length of the fragment $l$, the sliding offset ${\stackrel{⌢}{b}}_i$, the k-th fragment of $d_i$ can be defined as:

(2) $$s_i^k=[d_{i,k\ast bi},d_{i,k\ast bi+1},\dots ,d_{i,k\ast bi+l-1}]$$

It can be seen from Eq. (2) that $d_i$ is divided into $p_i/b_i$ overlap fragments by the adaptive sliding window. The reasons that the adjacent fragments overlap with each other are as follows. Firstly, it can increase the probability that the core fragment is only distributed in a fragment. Secondly, it can increase the variety of fragments, which can extract more features for the sequence. Thirdly, the fragment is not be dynamically masked in the feature extraction, so the overlap does not affect the model. Lastly, the number of fragments can be controlled by setting different ${\stackrel{⌢}{b}}_i$. This is important, as dimensions of inputs of many deep neural networks needs to be the same, such as the CNN and FCNN.

To make that numbers of fragments divided for different sequence are the same, ${\stackrel{⌢}{b}}_i$ is calculated by:

(3) $$b_i=p_i/K$$where K is the number of fragments. It can be seen from Eqs. (2) and (3) that regardless of the length of a sequence, the number of fragments divided by Eq. (3) is K.

After processing a SMILES by Eq. (2), low-level features of this drug can be extracted by:

(4) $${\stackrel{⌢}{O}}_i=[{\stackrel{⌢}{f}}_T({\stackrel{⌢}{s}}_i^1),{\stackrel{⌢}{f}}_T({\stackrel{⌢}{s}}_i^2),\dots ,{\stackrel{⌢}{f}}_T({\stackrel{⌢}{s}}_i^K)]$$where ${\stackrel{⌢}{s}}_i^1,{\stackrel{⌢}{s}}_i^2,\dots ,{\stackrel{⌢}{s}}_i^K$ are fragments of i-th drug divided by Eq. (2), ${\stackrel{⌢}{f}}_T$ is the trained RoBERTa model of the drug, ${\stackrel{⌢}{O}}_i\in R^{u\times K}$ is the extracted low-level features of i-th drug, ${\stackrel{⌢}{f}}_T({\stackrel{⌢}{s}}_i^k)$ is the output of ${\stackrel{⌢}{f}}_T$ whose dimension is $u\times 1$.

After processing a sequence of j-th target by Eq. (2), low-level features of this target can be extracted by:

(5) $${\stackrel{⌣}{O}}_j=[{\stackrel{⌣}{f}}_T({\stackrel{⌣}{s}}_j^1),{\stackrel{⌣}{f}}_T({\stackrel{⌣}{s}}_j^2),\dots ,{\stackrel{⌣}{f}}_T({\stackrel{⌣}{s}}_j^K)]$$where ${\stackrel{⌣}{s}}_j^1,{\stackrel{⌣}{s}}_j^2,\dots ,{\stackrel{⌣}{s}}_j^K$ are fragments of j-th target divided by Eq. (2), ${\stackrel{⌣}{f}}_T$ is the trained RoBERTa model of the target, ${\stackrel{⌣}{O}}_j\in R^{u\times K}$ is the extracted low-level features of j-th target.

Extracting features by Eqs. (4) and (5) has three advantages. Firstly, the extracted features own enough information to reconstruct the fragments. This is the property of the RoBERTa, as RoBERTa has a strong ability to fill the mask. Secondly, the extracted features own the relationships among fragments. As shown in Eqs. (4) and (5), ${\stackrel{⌢}{O}}_i$ and ${\stackrel{⌣}{O}}_j$ contain features of all fragments and the positional information of these fragments. Thirdly, the extracted features are the low-level features. RoBERTa is only used to extract the features for fragments, and which fragments and what relationships among fragments benefit for DTAP are further processed in the next steps. Therefore, the problem that the training goal of RoBERTa is very different from that of the DTAP can be overcome.

${\stackrel{⌢}{O}}_i$ and ${\stackrel{⌣}{O}}_j$ are low-level features, so another deep neural network should be designed to further extract the high-level features. Because fragments are divided by an adaptive sliding window (see Eq. (2)), the core fragment could be distributed in several adjacent fragments. Therefore, the deep neural network should be able to extract relationships among adjacent fragments. Furthermore, features of a fragment extracted by RoBERTa may also need to be weighted. As a result, a 2D CNN is further used to extract the high-level features.

RoBERTa is only used to learn the low-level features. Firstly, RoBERTa is a pre-trained model that requires a large number of training samples, but DTAP has fewer training samples, making it difficult to fully train the model. Secondly, it is difficult to define suitable tokens for RoBERTa that are used to obtain high-level features because species of ${\stackrel{⌢}{f}}_T({\stackrel{⌢}{s}}_i^2)$ and ${\stackrel{⌣}{f}}_T({\stackrel{⌣}{s}}_j^K)$ may be infinite, which may be hardly used as the token.

Given the low-level features ${\stackrel{⌢}{O}}_i$ and ${\stackrel{⌣}{O}}_j$ of the i-th drug and the j-th target, the high- level features can be extracted by:

(6) $${\stackrel{⌢}{h}}_i={\stackrel{⌢}{f}}_C({\stackrel{⌢}{O}}_i)$$

(7) $${\stackrel{⌣}{h}}_j={\stackrel{⌣}{f}}_C({\stackrel{⌣}{O}}_j)$$where ${\stackrel{⌢}{f}}_C$ and ${\stackrel{⌣}{f}}_C$ are two 2D CNN, ${\stackrel{⌢}{h}}_i$ and ${\stackrel{⌣}{h}}_j$ are 2D CNN-based high-level features of a drug and a target.

${\stackrel{⌢}{f}}_C$ and ${\stackrel{⌣}{f}}_C$ contain three 2D CNN layers. Each 2D CNN layer contains a 2D convolutional sublayer, a 2D batch normalization sublayer, and a rectified linear unit activation function. The reasons that only three 2D CNN layers are used here are as follows: Firstly, 2D CNN is only used for learning high-level features from low-level features, so it does not need too many layers. Secondly, there are too few labeled training samples, and 2D CNN with too many layers is easy overfitting.

The features extracted by Eqs. (6) and (7) have three advantages. Firstly, core fragments can be highlighted by the convolution kernel, as the fragment benefits for DTAP can get a larger response value. Secondly, the relationship benefit for DTAP can be extracted by the convolution kernel, as the convolution kernel has a strong ability to extract local positional relationships. Thirdly, meaningless positional information among fragments can be reduced by 2D CNN, as the field of view is getting bigger and bigger from the lower level to the upper level.

Features extracted by GCN and DTA score learned by FCNN

GCN is also used to extract high-level features for the drug. The reasons are as follows: Firstly, drug molecules have distinct graph structures, and the GCN can extract many useful features. Secondly, some special words are existed in SMILES, such as ‘.’, ‘\’, ‘/’, ‘7’, ‘8’, ‘6’, ‘)’, ‘5’ and ‘4’, which may affect the effectiveness of RoBERTa.

Given a drug $d_i=[d_{i,1},d_{i,2},\dots ,d_{i,pi}]$ whose length is p_i, a global pooling GCN (Zhang et al., 2021) is used to learn GCN based high-level features for the drug, which can be calculated by:

(8) $${\hat{h}}_i=\hat{f}G(d_i)$$where ${\hat{f}}_G$ is a GCN, which contains three GCN layers. For a GCN, the most crucial aspect is determining the graph vertices and edges. Similar to the approach in Zhang et al. (2021), the vertices in the graph representing a drug molecule correspond to atoms, and the features of these vertices are represented using one-hot encoding. The edges of the graph for a drug molecule are formed based on its chemical bonds, which are obtained utilizing the RDKit tool (RDKit: open-source cheminformatics, https://www.rdkit.org). If a bond exists between two atoms, it signifies the presence of an edge; otherwise, no edge is formed. Once the graph is constructed, PyTorch Geometric (https://pytorch-geometric.readthedocs.io/en/latest/) can be employed to run the GCN.

After extracting high-level features for a drug and a target by Eqs. (6)–(8), features of this drug-target pair can be concatenated by ${\stackrel{⌢}{h}}_i$, ${\stackrel{⌣}{h}}_i$ and ${\hat{h}}_i$ as follows:

(9) $$h_i=[{\stackrel{⌢}{h}}_i,{\stackrel{⌣}{h}}_i,{\hat{h}}_i].$$

The compatibility of ${\stackrel{⌢}{h}}_i$, ${\stackrel{⌣}{h}}_i$ and ${\hat{h}}_i$ in concatenating $h_i$ can be guaranteed for the following reasons. One is that the activation functions in 2D CNN and GCN can limit the output of neurons within a certain range. Another is that the last layer of our 2D CNN and GCN is a fully connected layer, which can adjust the output dimension.

The DTA score of the drug-target pair can be calculated as following:

(10) $$a_i=f_D(h_i)$$where ${\hat{f}}_D$ is a FCNN.

In the training, the DTA score of the drug-target pair is known, and ${\stackrel{⌢}{f}}_C$, ${\stackrel{⌣}{f}}_C$, ${\hat{f}}_G$, $f_D$ can be trained by the mean-square error between the prediction score and actual score.

The proposed ASSLDTA model

ASSLDTA consists of the ASSL training, supervised DTAP training, and supervised DTAP testing. ASSL training consists of step 1 and step 2. Supervised DTAP training consists of steps 3 to 9, where steps 3 and 4 can be concluded as ASSL-based feature extraction, step 5 is the 2D CNN-based high-level feature extraction, step 6 is the GCN-based high-level feature extraction, steps 7 to 9 can be concluded as DTA score learned by FCNN. These have been introduced in detail in “Parametric Analysis of the ASSLDTA” and “Results of DTA on Kiba”. Supervised DTAP testing consists of step 10 to step 15, which contain similar processes to supervised DTAP training. The code is available at https://github.com/yeqing0713/ASSLDTA.

In ASSLDTA, ASSL is used to learn the low-level features. Compared with other artificially defined low-level features, such as protein domains (Wang, Wu & Wang, 2024), motifs (Wang, Wu & Wang, 2024), recurrent neural network (RNN) (Hua et al., 2023), similarity matrixes (Ye, Zhang & Lin, 2022), 3D interaction information (Han, Kang & Guo, 2024), lots of traditional features (Kalemati, Zamani Emani & Koohi, 2024), and one-shot vector (Zhang et al., 2021), ASSL can utilize a large number of unlabeled training samples to learn more effective and comprehensive features. In ASSLDTA, two RoBERTa models are trained on fragments of drugs and targets and used to extract features for fragments. Compared with other pre-trained methods (Liu et al., 2019; Lennox, Robertson & Devereux, 2021; Elnaggar et al., 2020; Liu et al., 2021), ASSLDTA can overcome three problems of pre-trained methods mentioned in the “Problems of SSL usd in DTAP and Our Solution Strategy”. In ASSLDTA, ASSL together with 2D CNN is utilized. Compared with supervision methods (Öztürk, Özgür & Ozkirimli, 2018; Zeng et al., 2021; Wang, Wu & Wang, 2024; Aleb, 2021), ASSLDTA can extract low-level features by a large number of unlabeled samples, and extract high-level features that highlight the core fragments and their relationships by limited labeled samples.

Architectural parameter

ASSLDTA consists of two ASSL, two 2D CNN, a GCN, and an FCNN. Many architectural parameters exist in these neural networks, whose values are shown in Table 3.

ASSL primarily consists of RoBERTa. In ASSL, the number of attention heads and the number of hidden layers of RoBERTa are respectively set to 6 and 3, as the numbers of tokens are only 62 and 25 in the drug and target, which makes that a small RoBERTa model has enough power to extract features for these tokens. The hidden size of RoBERTa is set to 60, as this value must be a multiple of the number of attention heads. The pooler fully connected size is set to 60, which determines the dimension u of ${\stackrel{⌢}{O}}_i$ and ${\stackrel{⌣}{O}}_j$. The u cannot be too big, as setting a larger input size for 2D CNN usually requires deeper networks or larger convolution kernels, which could be overfitting when training samples are not enough. Batch size is set to 128, as our model is smaller than the default model and can be trained by more samples in the same time. Other parameters are set to the default values in the Hugging Face toolbox (Transformer, https://github.com/huggingface). Since ASSL is a pre-trained model that requires extensive training time and is only used for extracting low-level features, the parameters have a limited impact on the experimental results within a wide range. Therefore, most of the ASSL parameters are set to fixed values based on the above analysis.

In the 2D CNN, the numbers of filters for three 2D CNN layers are respectively set to 64, 128, and 256. A total of 256 is set for the third 2D CNN layer, as the dimension of the output can be 2560, which is a bit larger than the dimension of the output of the FC layer 1-2. The kernel sizes are respectively set to 5, 3, and 3, as some useful features should be extracted in a larger field of view in the first layer. Dimensions of the output of FC layer 1-1 and FC layer 1-2 are set to 1,024 and 2,048, as the high-level features of the drug are extracted by two methods and that of the target is only extracted by a method.

The reasons of setting parameters for the GCN, FCNN, and global parameters are similar to that for the GCN and FCNN in Zhang et al. (2021).

Experimental setting

Five datasets listed in Table 1 are used to validate the method, such as Kiba, Davis, DTC, Metz, and Tox-Cast. The Human, C.elegans, and BindingDB datasets are used to validate the effectiveness of CPI. To verify generalization ability, our experiments split the dataset into training, validation, and test sets in a ratio of 8:1:1. Five-fold cross-validation was used, and the process was executed five times. The number of samples in the training, validation, and test sets for each dataset are presented in Table 4.

The concordance index (CI), mean squared error (MSE), $r_m^2$ and area under the precision-recall curve (AUPR) are used as the metrics for DTA, where CI measures the rank of the predicted binding affinity (Mukherjee, Ghosh & Basuchowdhuri, 2022), and MSE measures the difference between the vector of predicted values and the vector of the actual value (Zhang et al., 2021), $r_m^2$ calculated as the ratio of the sum of squared differences between the predicted values and the mean of the dependent variable to the total sum of squared differences between the actual values and the mean of the dependent variable (Xia et al., 2023b), AUPR measures the area under the curve that plots precision against recall (Ye, Zhang & Lin, 2023). The area under the ROC curve is used as the metric for CPI, where AUROC is the area under the receiver operating characteristic curve (Ye, Zhang & Lin, 2022). Precision is used as the metric for the fill masking of ASSL. Let P and N are the numbers of positive samples and negative samples of a dataset. TP and TN denote numbers of true positives and true negatives. FP and FN denote numbers of false positives and false negatives. Precision can be defined as:

(11) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{TP}{TP+FP}}$$

Parametric analysis of the ASSLDTA

It can be seen from Algorithm 1 that three parameters should be set for ASSLDTA, such as $\stackrel{⌢}{l}$, $\stackrel{⌣}{l}$ and K. They are set according to the lengths of SMILESs and targets on five DTA datasets, which are presented in Figs. 1 and 2.

It can be seen from Fig. 1 that most lengths of SMILESs are distributed in 14 to 75, and lower quartile lengths of SMILESs are respectively 39, 45, 41, 37, and 18 on five DTA datasets. These lengths are not very larger than the lengths of sentences in the text. As a result, $\stackrel{⌢}{l}$ should be set to the value that the fragment with this length contains enough information for DTA. In our experiment, $\stackrel{⌢}{l}$ is set to 16, as 16 is slightly larger than 14, which makes that most core fragments can appear in one fragment and numbers of fragments divided for most SMILESs are not too big.

It can be seen from Fig. 2 that most lengths of sequences are distributed in 215 to 1,732, and lower quartile lengths of SMILESs are respectively 450, 479, 479, 464, and 407. These lengths are very larger than the lengths of sentences in the text. As a result, there are three factors to consider when setting $\stackrel{⌢}{l}$, such as the lengths of the core fragments, the model processing power, and the number of fragments divided for most sequences. In our experiment, $\stackrel{⌣}{l}$ is set to 128, which is slightly larger than the half of 215.

In fact, $\stackrel{⌢}{l}$ and $\stackrel{⌣}{l}$ can be set in a wide range, as lengths of core fragments vary over a wide range. Specifically, if the core fragment is spread across multiple fragments, features of this core fragment can be further extracted by the 2D CNN. If the core fragment is too short, features of this core fragment can be further extracted by weighting features of the corresponding fragment, which also can be extracted by 2D CNN. Furthermore, the RoBERTa needs to be trained on large-scale data, so $\stackrel{⌢}{l}$ and $\stackrel{⌣}{l}$ are hardly set by the experiment. As a result, $\stackrel{⌢}{l}$ and $\stackrel{⌣}{l}$ are simply set to 16 and 128 according to above analyses but not according to the experiment.

Next, the principle of setting K is introduced. K should be set to the value that most words are distributed in one or more fragments. Therefore, $K\ast \stackrel{⌢}{l}$ and $K\ast \stackrel{⌣}{l}$ should be respectively larger than 81 and 1,732, which are the upper limit of most drugs and targets presented in Figs. 1 and 2. Furthermore, some overlap should exist among the neighbor fragments. As a result, K should be larger than 13.5 which is 1,732/128, and results of ASSLDTA whose K is set to 16, 24, 32 are presented in Table 5, where the best of each group on each dataset is shown in bold.

It can be seen from Table 5 that CIs, MSE and AUPR of ASSLDTA are the best when K is set to 24, and $r_m^2$ of ASSLDTA are the best on most datasets when K is set to 16. Furthermore, results of ASSLDTA when K is set to 32 are much worse than results of ASSLDTA when K is set to 16 and 24, the reason may be that low-level features extracted by ASSL contain too much positional information when K is too large. As a result, K is set to 24 in the next experiments, as K set to 24 is the balance for the number of the fragments and the positional information. Additionally, Table 5 also indicates that adjusting K within a wide range does not significantly impact the results. The primary reason is that K only affects the degree of window overlap, and the subsequent 2D CNN can adapt to different degrees of window overlap to a certain extent. We set the same K value for all databases, which to some degree demonstrates the good generalization ability of ASSLDTA.

Results of DTA on Kiba

In this subsection, ASSLDTA is compared with many recent works on Kiba, such as DeepDTA (Öztürk, Özgür & Ozkirimli, 2018), MATT (Zeng et al., 2021), WideDTA (Öztürk, Ozkirimli & Özgür, 2019), MAM (Aleb, 2021), AttentionDTA (Zhao et al., 2019), SAG-DTA (Zhang et al., 2021), GraphDTA (Nguyen, Le & Venkatesh, 2021), MGraphDTA (Pan et al., 2023), DeepGS (Lin, 2020), DGraphDTA (Chen et al., 2020), GLCNDTA (Gao et al., 2018), TDGraphDTA (Zhu et al., 2023a), DeepGLSTM (Mukherjee, Ghosh & Basuchowdhuri, 2022), GCN-BERT (Lennox, Robertson & Devereux, 2021), and DGDTA (Yang et al., 2022). To comprehensively consider the role of each module in ASSLDTA, ASSLDTA is also compared with MAM, 2DCNN-GCN, 2DCNN-BERT, MPBERT and CPBERT, where the backbone network of MAM is 2DCNN, the network structure of 2DCNN-GCN is similar with ASSLDTA but its input is one-hot vector, the input of 2DCNN-BERT is consistent with ASSLDTA but it does not have GCN. MPBERT employees Mol-BERT (Xia et al., 2023b) to extract features for drugs, while ProteinBERT (Brandes et al., 2022) is utilized to capture features for targets. CPBERT employees ChemBERTa-2 (Ahmad et al., 2022) to extract features for drugs, while ProteinBERT (Brandes et al., 2022) is utilized to capture features for targets. The model evaluations of the DTA on Kiba are listed in Table 6, where the best of each group on each dataset is shown in the bold font. It can be seen from Table 6 that ASSLDTA is the best among all compared methods on Kiba.

Firstly, ASSLDTA is much better than all CNN-based methods. The CI, MSE, $r_m^2$ and AUPR of ASSLDTA are much higher than that of the compared CNN-based methods. Specifically, the CI of ASSLDTA is 0.01 higher than the CI of MAM, and the MSE of ASSLDTA is 0.011 less than the MSE of MAM, where MAM is the best method among CNN-based methods. They prove that using ASSL to learn low-level features for drugs and targets is better for DTAP.

Secondly, ASSLDTA is much better than all GCN- and CNN-based methods. The CI, MSE, $r_m^2$ and AUPR of ASSLDTA are much higher than that of the compared GCN- and CNN-based methods. Specifically, the CI of ASSLDTA is 0.004 higher than the CI of DGraphDTA, and the MSE of ASSLDTA is 0.002 less than the MSE of DGraphDTA, where DGraphDTA is the best method among GCN- and CNN-based methods. They prove that using ASSL together with 2D CNN to learn high-level features is better.

Thirdly, ASSLDTA is much better than other pre-trained model-based methods. There are seven pre-trained models for comparison, including DeepGLSTM, GCN-BERT, SubMDTA, TransVAEDTA, 2DCNN-BERT, MPBERT, and CPBERT. The CI, MSE, $r_m^2$ and AUPR of ASSLDTA are much higher than that of other pre-training model-based methods. Specifically, the CI, MSE, $r_m^2$ and AUPR of ASSLDTA are 0.004, −0.004, 0.019 and 0.016 higher than the CI of CPBERT, where CPBERT is the best method among the pre-trained model-based methods by using different pre-training methods. They prove that the effectiveness of our improved RoBERTa for SMILES data and protein sequences compared to other architectures. The possible reason is as follows: other methods use the entire drug and target as training data, which may obscure the fragment features that play a crucial role in drug-target affinity. In contrast, our pre-training approach focuses on feature extraction from fragments, enabling better preservation of local fragment features favorable to drug-target affinity, thus facilitating the subsequent application of supervised methods for highlighting these features.

Fourthly, the module of BERT is the most important in ASSLDTA. It can be seen from Table 6 that the CI of ASSLDTA is respectively 0.010, 0.006, 0.01 0.005 and higher than that of 2DCNN-BERT, 2D-CNN-GCN, MAM and GBERT. The MSE of ASSLDTA is 0.002, 0.008, 0.012 and 0.006 less than that of 2DCNN-BERT, 2D-CNN-GCN and MAM. Furthermore, $r_m^2$ and AUPR of ASSLDTA are also much higher than these ablation methods. They demonstrate that ASSL can significantly enhance the performance of DTAP, while GCN has limited impact on enhancing DTAP.

Results of DTA on Davis

In this subsection, ASSLDTA is compared with many recent works on Davis, such as DeepDTA (Öztürk, Özgür & Ozkirimli, 2018), MATT (Zeng et al., 2021), WideDTA (Öztürk, Ozkirimli & Özgür, 2019), MAM (Aleb, 2021), AttentionDTA (Zhao et al., 2019), SAG-DTA (Zhang et al., 2021), GraphDTA (Nguyen, Le & Venkatesh, 2021), MGraphDTA (Pan et al., 2023), DeepGS (Lin, 2020), DGraphDTA (Chen et al., 2020), GLCNDTA (Gao et al., 2018), TDGraphDTA (Zhu et al., 2023a), DeepGLSTM (Mukherjee, Ghosh & Basuchowdhuri, 2022), GCN-BERT (Lennox, Robertson & Devereux, 2021), and DGDTA (Yang et al., 2022). To comprehensively consider the role of each module in ASSLDTA, ASSLDTA is also compared with MAM, 2DCNN-GCN, 2DCNN-BERT, MPBERT and CPBERT, where the backbone network of MAM is 2DCNN, the network structure of 2DCNN-GCN is similar with ASSLDTA but its input is one-hot vector, the input of 2DCNN-BERT is consistent with ASSLDTA but it does not have the GCN. GBERT employees Mol-BERT (Xia et al., 2023b) to extract features for drugs, while ProteinBERT (Brandes et al., 2022) is utilized to capture features for targets. CPBERT employees ChemBERTa-2 (Ahmad et al., 2022) to extract features for drugs, while ProteinBERT (Brandes et al., 2022) is utilized to capture features for targets. The model evaluations of the DTA on Davis are listed in Table 7, where Davis was often used to evaluate DTAP. It can be seen from Table 6 that ASSLDTA is the best among all compared methods on Davis.

Firstly, ASSLDTA is much better than all CNN-based methods. The CI, MSE, $r_m^2$ and AUPR of ASSLDTA are much higher than that of the compared CNN-based methods. Specifically, the CI of ASSLDTA is 0.017 higher than the CI of AttentionDTA, and the MSE of ASSLDTA is 0.017 less than the MSE of AttentionDTA, where AttentionDTA is the best method among CNN-based methods. They prove that using ASSL to learn low-level features for drugs and targets is better for DTAP.

Secondly, ASSLDTA is much better than all GCN- and CNN-based methods. The CI, MSE, $r_m^2$ and AUPR of ASSLDTA are much higher than that of the compared GCN- and CNN-based methods. Specifically, the CI of ASSLDTA is 0.004 higher than the CI of TDGraphDTA, where TDGraphDTA is the best method among GCN- and CNN-based methods. They prove that using ASSL together with 2D CNN to learn high-level features is better.

Thirdly, ASSLDTA is much better than other pre-trained model-based methods. There are seven pre-trained models for comparison, including DeepGLSTM, GCN-BERT, SubMDTA, TransVAEDTA, 2DCNN-BERT, MPBERT, and CPBERT. The CI, MSE, $r_m^2$ and AUPR of ASSLDTA are much higher than that of other pre-training model-based methods. Specifically, the CI, MSE, $r_m^2$ and AUPR of ASSLDTA are 0.007, −0.002, 0.009 and 0.008 higher than the CI of CPBERT, where CPBERT is the best method among the pre-trained model-based methods by using different pre-training methods. They prove that the effectiveness of our improved RoBERTa for SMILES data and protein sequences compared to other architectures. The possible reason is that other methods use the entire drug and target as training data, which may obscure the fragment features that play a crucial role in drug-target affinity. In contrast, our pre-training approach focuses on feature extraction from fragments, enabling better preservation of local fragment features favorable to drug-target affinity, thus facilitating the subsequent application of supervised methods for highlighting these features.

Fourthly, the module of BERT is the most important in ASSLDTA. It can be seen from Table 7 that the CI of ASSLDTA is respectively 0.002, 0.006, 0.019 and 0.008 higher than that of 2DCNN-BERT, 2DCNN-GCN, MAM and GBERT. The MSE of ASSLDTA is 0.002, 0.007 and 0.003 less than that of 2DCNN-BERT, 2DCNN-GCN and GBERT. They demonstrate that ASSL can significantly enhance the performance of DTAP, while GCN has limited impact on enhancing DTAP.

Furthermore, although the MSEs of GCN-BERT and MAM are better than that of ASSLDTA on Davis, MSEs, the CIs of GCN-BERT and MAM are much worse than those of ASSLDTA on Kiba, and the CIs of GCN-BERT and MAM are much worse than those of ASSLDTA on Davis. They prove that GCN-BERT and MAM are not stable and ASSLDTA are also much better than GCN-BERT and MAM.

Results of DTA on other three datasets

In this subsection, ASSLDTA is compared with GraphDTA and DeepGLSTM on DTC, Metz, and Tox-cast. Only GraphDTA and DeepGLSTM are compared, as few deep methods are currently validated on these three databases. Results are presented in Figs. 5 and 6.

It can be seen from Fig. 5 and 6 that our method is also the best. Firstly, the CI of ASSLDTA is respectively 0.025, 0.015, and 0.008 higher than that of GraphDTA, and the MSE of ASSLDTA is respectively 0.034, 0.038, and 0.027 less than that of GraphDTA on DTC, Metz, and Tox-Cast. Secondly, the CI of ASSLDTA is respectively 0.006, 0.005, and 0.004 higher than that of GraphDTA and the MSE of ASSLDTA is respectively 0.007, 0.015, and 0.016 less than that of GraphDTA on DTC, Metz, and Tox-Cast. They prove that using ASSL together with 2D CNN to learn high-level features for the drug and target is better for DTAP and demonstrates the good generalization ability of ASSLDTA.

Results of CPI

In this subsection, the binary classification task of CPI is also used to evaluate the ASSLDTA. ASSLDTA is compared with MGraphDTA (Wang et al., 2024), GraphDTA (Nguyen, Le & Venkatesh, 2021), SAG-DTA (Zhang et al., 2021), SVM (Chen et al., 2020), TransformerCPI (Chen et al., 2020), GNN-CNN (Zhao et al., 2024), and TrimNet (Li et al., 2021). The results are presented in Table 8, where our method is shown in bold.

It can be seen from Table 8 that AUROC of ASSLDTA is the best on all datasets. AUROCs of ASSLDTA are respectively 0.005, 0.003 and 0.006 higher than that of SAG-DTA, MGNN-MCNN and SAG-DTA on the Human, C.elegans and BindingDB datasets, where SAG-DTA, MGraphDTA and SAG-DTA are respectively the second best methods on these three datasets. They prove that ASSLDTA is also benefit for CPI, which also prove the good generalization ability of ASSLDTA.

Fill mask evaluations of the ASSL

In this subsection, the filling mask task on five DTA datasets is used to evaluate the effectiveness of ASSL. In the padding mask task, the second to last character of a SMILES or a sequence is masked sequentially. The RoBERTa model is then used to predict the masked characters. The more accurate the prediction of the character, the better the model can extract the features of the drug or target. The predicted precisions of the characters are presented in Figs. 7 and 8, where the characters around the perimeter have been sorted according to the top-1 precisions. The overall prediction precisions for the masked characters are presented in Figs. 9 and 10. Top-p indicates that the masked character is in the set of characters with top-p prediction scores.

ASSL trained on drug fragments has a strong ability to reconstruct drug fragments. It can be seen from Fig. 7 that the Top-3 precisions of most characters are greater than 0.6. Specifically, the Top-10 precisions of most characters are greater than 0.95. It can be seen from Fig. 9 that the overall Top-1, Top-3, Top-5 and Top-10 precisions of the SMILES are respectively larger than 0.624, 0.855, 0.917, and 0.967, where the number of different characters is greater than 28. They support the validity of the features extracted by ASSL for drugs.

The ASSL trained on fragments of the target also has a strong ability to reconstruct the fragment of the target. It can be seen from Fig. 8 that the Top-1 precisions of most characters are greater than 0.4. Specifically, the Top-10 precisions of most characters are greater than 0.80. It can be seen from Fig. 10 that the overall Top-1, Top-3, Top-5, and Top-10 precisions of the sequence are respectively larger than 0.508, 0.696, 0.780, and 0.905, where the number of different characters is greater than 20. They support the effectiveness of the features of the target extracted by the ASSL.

Furthermore, it can be seen from Figs. 7–10 that the prediction precisions of SMILESs are better than that of the sequence. However, in ASSLDTA, GCN is also used to extract features for SMILES. The reasons are as follows: Firstly, the Top-1 precisions of many characters are very low, which may have a negative impact on the quality of drug feature extraction. It can be seen from Fig. 7 that the Top-1 precisions of about 1/3 characters are smaller than 0.4, such as ‘.’, ‘\’, ‘/’, ‘7’, ‘8’, ‘6’, ‘)’, ‘5’ and ‘4’. Secondly, the molecular structure of a drug is a typical graph structure, and then effective drug features can be extracted by using a GCN.