Double-target self-supervised clustering with multi-feature fusion for medical question texts

Text representation module

Lexical semantic features

The fastText model involves input layer, hidden layer and output layer. The model is developed on the basis of skip-gram, and each word of the input context is decomposed based on word n-gram format. It can also represent the internal order of words. Take the phrase “primary liver cancer” as an example. If the value of n-gram is 2, then its bigram include “<primary”, “primary liver”, “liver cancer”, and “cancer>”. The word vector of “primary liver cancer” can be represented by superimposing the 4 word vectors after decomposition. The lexical semantic features generated by the fastText model contain semantic and partial sequential information. The question text in the medical question-and-answer community has very few words, and the contextual semantics and word order information of the feature word are obviously missing. The lexical semantic features are extracted by the fastText model, and lexical semantic features of the medical question text D are represented as: (1)${\displaystyle D=\left(\begin{array}{ccccc}{a}_{11}\hfill & a_{12}\hfill & a_{13}\hfill & \cdots \hfill & a_{1T}\hfill \\ a_{21}\hfill & a_{22}\hfill & a_{23}\hfill & \cdots \hfill & a_{2T}\hfill \\ ⋮\hfill & ⋮\hfill & ⋮\hfill & \cdots \hfill & ⋮\hfill \\ a_{k1}\hfill & a_{k2}\hfill & a_{k3}\hfill & \cdots \hfill & a_{kT}\hfill \end{array}\right)}$

where a_k denotes the vector of word w_k in the question text D and T is the vector dimension.

Weighted lexical semantic features

The TF-IDF values are used to weight the fastText vocabulary semantics, so as to reduce the influence of non-keyword features and improve the ability to distinguish topics. Weight features of the vocabulary in question text D are shown below: (2)${\displaystyle W=\left[tfid{f}_1,tfid{f}_2,\dots ,tfid{f}_k\right]}$

where tfidf_k denotes the weight of the word w_k in the question text D. A bigger TF-TDF value means more importance.

We multiplied the lexical semantic features of the text with the corresponding TF-TDF values, and obtained the weighted lexical semantic features FW of the question text D, with FW ∈ ℝ^k×T. (3)${\displaystyle FW=WF=\left(\begin{array}{ccccc}{a}_{11}\times tfid{f}_1\hfill & a_{12}\times tfid{f}_1\hfill & a_{13}\times tfid{f}_1\hfill & \cdots \hfill & a_{1T}\times tfid{f}_1\hfill \\ a_{21}\times tfid{f}_2\hfill & a_{22}\times tfid{f}_2\hfill & a_{23}\times tfid{f}_2\hfill & \cdots \hfill & a_{2T}\times tfid{f}_2\hfill \\ ⋮\hfill & ⋮\hfill & ⋮\hfill & \cdots \hfill & ⋮\hfill \\ a_{k1}\times tfid{f}_k\hfill & a_{k2}\times tfid{f}_k\hfill & a_{k3}\times tfid{f}_k\hfill & \cdots \hfill & a_{kT}\times tfid{f}_k.\hfill \end{array}\right)}$

Auto-encoder module

BiGRU

The encoder is essentially a RNN that encodes the input sequence into a feature representation. For lexical sequence prediction, an arbitrary question text sequence $\mathrm{X}=\left({\mathrm{x}}_1,{\mathrm{x}}_2,\dots ,{\mathrm{x}}_{\mathrm{k}}\right)$, x_i ∈ ℝ^T is given, where k is the number of words in the question text and T is the dimension of the pre-trained word embedding, i.e., the topic dimension. Then, the encoder was used to learn the mapping h_t = f₁(h_t−1, x_t)from x_t to h_t at time step t, where h_t ∈ ℝ^m is the hidden state of the encoder at time t, m is the size of the hidden state, and f₁ is a nonlinear function. In the study, BiGRU was used as f₁ to obtain long-term dependent neuron information. Another reason for using GRU units was that the neurons sum over a certain time, which helps to overcome the problem of gradient disappearance and better capture long-term correlations in time series. Each GRU unit consists of two Sigmoid gates, namely the reset gate r_t and the update gate z_t . The GRU unit is updated as follows: (4)${\displaystyle r_t=\sigma \left(W_r\left[h_{t-1},x_t\right]\right)}$ (5)${\displaystyle z_t=\sigma \left(W_z\left[h_{t-1},x_t\right]\right)}$ (6)${\displaystyle {\tilde{h}}_t=tan\mathrm{h}\left(W_{\tilde{h}}\left[r_t{h}_{t-1},x_t\right]\right)}$ (7)${\displaystyle h_t=\left(1-z_t\right)h_{t-1}+z_t{h}_{\tilde{t}}}$ (8)${\displaystyle y_t=\sigma \left(W_o{h}_t\right)}$

where [h_t−1,x_t] ∈ ℝ^(m+T) is the connection of the previous hidden state h_{t −1} and the current input x_t.,; $x_t\cdot W_r,W_z,W_{\tilde{h}}\in R^{m\times \left(m+\mathrm{T}\right)}$ are parameters to be learned; σ is the sigmoid function. A bidirectional structure was used to obtain the dependencies between adjacent words in a single question text. (9)${\displaystyle \overrightarrow{h_t}=\overrightarrow{GRU}\left(W_t,\overrightarrow{h_{t-1}}\right)}$ (10)${\displaystyle \overleftarrow{h_t}=\overleftarrow{GRU}\left(W_t,\overleftarrow{h_{t-1}}\right)}$ (11)${\displaystyle h_t=\left[\overrightarrow{h_t};{\overleftarrow{h}}_t\right].}$

Finally $\overrightarrow{h_t}$ and ${\overleftarrow{h}}_t$are horizontally spliced to obtain a hidden state h_t.. Let the number of hidden units per one-way GRU be u. For simplicity, all h_t is H ∈ ℝ^k×2u, where k is the number of words in the input question text.

Self-attention mechanism

The self-attention mechanism is a variation of the attention mechanism. Unlike the attention mechanism, the encoder and decoder of the self-attention mechanism deal with the same text. According to the word distribution in the text, the mechanism calculates the weight of each word, i.e., the interaction between words, which reduces the reliance on external information and is better at capturing the internal relevance of data or features. Each feature was assigned different weights, with complementary feature information enhanced and the conflicting parts weakened. A common approach in many previous studies has been to use the final hidden state of the RNN or to establish the simple vector representation through the max-pooling (or mean-pooling) of RNN hidden state. However, it is relatively difficult to have multiple semantics in all time steps of an RNN model and there are lexical semantics that are not necessary. Different from previous approaches, a self-attention mechanism was introduced which allowed different aspects of a sentence to be extracted into multiple vector representations. In the proposed sentence embedding model, a self-attention mechanism was introduced in BiGRU to obtain more information without other additional input. It relieved the encoder of some long-term memory burden due to the direct access to the hidden representation at all times. The computational procedure of the self-attention layer is as follows: (12)${\displaystyle Z=W_2\cdot tanh\left(W_1\cdot H^T\right)}$ (13)${\displaystyle A=\frac{exp\left(Z\right)}{\sum exp\left(Z\right)}}$ (14)${\displaystyle \hat{\mathrm{Z}}=\mathrm{A}\cdot {\mathrm{H}}^{\mathrm{T}}}$ where W₁ andW₂ are linear layers with learnable parameters; tanh is the activation layer with nonlinear transformation; H is the output feature of GRU; Z is the linear layer output feature. The self-attention matrix A was computed by the Softmax function. Exp represents the exponential function, and each value in A represents the correlation for the feature value with other feature values. The enhanced intermediate feature $\hat{Z}$ on the basis of the self-attention mechanism was obtained by multiplying the attention matrix A with the input H. Then the output features are obtained through the linear projection layer calculation of $\hat{Z}$, which is illustrated as follows: (15)${\displaystyle M=W_{2}tanh\left(W_1^T\hat{Z}\right)}$

where W₁ and W₂ are linear layers with learnable parameters; tanh is the activation layer with nonlinear transformation. M ∈ R^1×r is the final text representation vector, where r is the vector dimension.

Double-target auxiliary module

To construct a model of end-to-end feature extraction text clustering and incorporate cross-document topic information to obtain a more friendly clustering representation, we introduced the target loss function L_topic for cross-document topic information, fused the clustering target loss function L_cluster into the network loss function of the learning representation, and constructed the joint loss function in a unified framework. In this way, the text representation was fused with clustering results under a unified framework.

The cross-document topic target loss function L_topic. To further learn cross-document-topic information, the document-topic distribution T was used as the self-supervised training target. Under the assumption that there was one-to-one mapping relationship between document clusters and topics, it enabled the auto-encoder BiGRU to learn a relatively clustering-friendly representation with the cross-document information in the training phase. However, it was necessary to obtain a prediction vector with the same dimensionality as the corresponding document-topic vector. We used a two-layer learnable fully-connected layer to reduce the dimensionality of the text representation vector M.

The Softmax function was used to calculate the posterior probability distribution of the text representation M, as illustrated in the following formula. (16)${\displaystyle D=Softmax\left(W_2\mathrm{Re}lu\left(W_1^{T}M\right)\right)}$

where D ∈ R^1×r; r is the length of the document-topic distribution vector; ReLU is a linearly corrected activation function that serves to set input negative values to zero; W₁ andW₂ are linear layers with learnable parameters for the dimensionality reduction of the text representation vector M, which make the dimensionality of M the same as that of the document-topic vector.

The result d_ij ∈ D represents the probability that the sample i belongs to the topic j and D is a probability distribution. The final loss function can be defined as follows: (17)${\displaystyle L_{\text{topic}}=KL\left(T\left|\right|D\right)=\sum _i\sum _j{t}_{ij}log\frac{t_{ij}}{d_{ij}}.}$

T is the document-topic distribution of the text computed by the topic model LDA, i.e., the probability distribution of each sentence related to the topic. The KL-divergence was used as the training model of the loss function.

The clustering target loss function L_cluster. To construct end-to-end text clustering models and enable auto-encoder BiGRU to learn a friendly clustering representation, we proposed a batch-based clustering loss to improve the clustering of output features, which was inspired by the research of Xie, Girshick & Farhadi (2016) and Wen et al. (2016). This loss is represented as follows: (18)${\displaystyle L_{cluster}=\sum _{i=1}^m\sum _{k=1}^K{\left(x_{i,k}-c_k\right)}^2}$

where m is the number of samples within each batch; K is the number of clustering centroids within each batch, i.e., the number of clustering categories 12; c is the clustering centroid of each batch; x is the samples within each batch. In each training iteration, the clustering centroids c was obtained by clustering the features within the batch through the K-Means algorithm. L_cluster made the features within the batch close to the clustering centroid c.

The joint target loss function L_train. Combining the two target functions designed, we obtained the overall loss of the training phase, which is shown as follows. (19)${\displaystyle L_{\text{train}}=\alpha L_{cluster}+L_{topic}}$

where α is the hyperparameter. The loss L_train was continuously being optimized during the clustering, updating the model parameters. In the early iterations of the algorithm, the features extracted by the model were not yet ideal, and the L_cluster with a too large α might produce side effects. Hence we used a simple climbing strategy to adjust α, so that it gradually increased with the training iterations proceeding, as shown in the following formula: (20)${\displaystyle \alpha \left(t\right)=\frac{t}{T}\ast {\alpha }_{max},t\le T^{{}^{\prime }}}$

where α_max = 0.3; t is the number of iterations; and T’ is the fixed number of iterations.