Unsupervised online multitask learning of behavioral sentence embeddings

Sequence-to-sequence sentence embeddings

The sequence-to-sequence model maps input sequences to output sequences using an encoder–decoder architecture. Given an input sentence x = (x₀, x₂, …, x_T) and output sentence y = (y₀, y₂, …, y_T′), where x_t and y_t represent individual words, the standard sequence model can be expressed as computing the conditional probability (1)${\displaystyle P\left(\mathbf{y}|\mathbf{x}\right)=\prod _{t=0}^{T^{\prime }}P\left(y_t|y_{i<t},\mathbf{s},h\right)}$

where s is the sequence of outputs s_t from the encoder and h is the internal representation of the input given by the last hidden state of the encoder. For a given dataset $\mathcal{D}={\left\{\left({\mathbf{x}}_n,{\mathbf{y}}_n\right)\right\}}_{n=1}^N$, we denote the learned internal representation as ${\displaystyle h_{\theta }\equiv f\left(\mathbf{x}|\mathcal{D}\right)=f\left(\mathbf{x}|\theta \right)}$ where f(⋅) is the encoder function and θ is the set of parameters resulting from $\mathcal{D}$. The internal representation h_θ encodes the input x into a vector space that allows the decoder to generate a good estimate of y. In cases where $\mathcal{D}$ contains semantically-related data pairs, h_θ can be viewed as a semantic vector representation of the input, or sentence embedding, which can be useful for subsequent NLP tasks. In our case we apply contextual learning and designate consecutive sentences in continuous corpora as x and y.

While this model allows us to obtain semantically rich embeddings through training on unlabeled data, the quality of the embeddings is highly influenced by biases in the data and prevents the embeddings from becoming specialized in any target task (Conneau et al., 2017). Therefore we propose to enhance the quality of unsupervised sentence embeddings through multitask learning.

Multitask embedding training

The addition of a multitask objective can guide embeddings into a space that is more discriminative in a target application. We hypothesize that this holds true even when the multitask labels are generated online from unlabeled data with no assumption of label reliability, as long as there is some relation between the multitask and target application.

Assuming an online system which generates multitask labels b for each input x we can augment the dataset to yield ${\mathcal{D}}_{\mathrm{aug}}={\left\{\left({\mathbf{x}}_n,{\mathbf{y}}_n,{\mathbf{b}}_n\right)\right\}}_{n=1}^N$. We then aim to predict this new label b in conjunction with the original output sequence y. This is implemented in our seq2seq model by adding another head to the internal representation h, shown in Fig. 1, which we will refer to as the multitask network. In addition to Eq. (1), the model now also estimates the conditional probability ${\displaystyle P\left(\mathbf{b}|\mathbf{x}\right)=g\left(h|{\mathcal{D}}_{\mathrm{aug}}\right)=g\left({h_{\theta }}_{\mathrm{aug}}\right)}$

where g(⋅) is the network function for online transfer learning using the multitask network and h_θaug is the new internal representation given by ${\mathcal{D}}_{\mathrm{aug}}$. In this work, g(⋅) is implemented using a multilayer perceptron. The overall architecture is shown in Fig. 1.

The training loss is then the weighted sum of losses from the multiple tasks, defined as ${\displaystyle J=\lambda \cdot L_1\left(\mathbf{y},\mathbf{x}\right)+\left(1-\lambda \right)\cdot L_2\left(\mathbf{b},\mathbf{x}\right)}$ where L₁ and L₂ are the cross entropy losses for contextual learning and the additional task, respectively. With most multitask setups there is an issue on how to control the training ratio λ to account for different data sources. For example, if there is no overlap in inputs of the multiple tasks then λ can only alternate between 0 and 1 during training to switch between the different tasks. However, since we propose a multitask objective whose labels are generated from incoming data we are able to freely adjust λ. It is possible to adjust the multitask ratio as training progresses to put emphasis on different tasks but we do not make any assumptions on the optimal weighting scheme and give equal importance to both tasks by setting λ to 0.5.

Online multitask label generation

To guide the embeddings in becoming more suitable for affective tasks, we select a multitask objective that classifies the polarity in sentiment (positive or negative) of input sentences. Tasks such as emotion recognition or human behavior analysis (Narayanan & Georgiou, 2013) are more complicated than these two affective states, however we hypothesize this is a related task allowing for domain knowledge transfer into the sentence embeddings.

We generate the affective labels for each input during training using an online mechanism. In our online approach we apply the simplest method by automatically labeling inputs using a simple, knowledge-driven, look-up table of likely affect of single words (Tausczik & Pennebaker, 2010). Specifically, we use words categorized in the two top-level affective states: negative and positive emotion. An input sentence is assigned a Negative or Positive label based on the majority number of words corresponding to each affective state. Some examples of affective words in the affective look-up table are shown in Table 1.

Evidently, this labeling approach differs slightly from sentiment analysis (Pang & Lee, 2008), which mostly focuses on classifying the polarity of subjective opinions. In our case we label all the inputs naively based on the count of affective words and do not consider semantic context or even simple word negation. We expect this approach to deviate greatly from the ground truth, and that truth may be contextual, subjective, and fluid, however we hypothesize the inclusion of affective knowledge in embeddings will still be beneficial in identifying more complex behaviors or emotions later. Specifically, we do not want to constrain the system through methods such as (Pang & Lee, 2008) but rather place emphasis and focus on domain relevant terms.

Unsupervised clustering of embeddings

As an initial evaluation step we analyzed the performance of the embeddings on a binary behavior classification task using minimal training on the Couples Therapy Corpus which will be described below. We applied a simple k-means clustering method on sentence embeddings from training sessions to obtain two clusters. We then labeled the clusters by randomly selecting a single session from the training set as seed and assigning the session label to the cluster which the majority of embeddings in that session belonged to. The other cluster was subsequently labeled as the opposite class label. Final test session labels were predicted based on which cluster the majority of embeddings from a session were in. Although this method of behavior classification is very rudimentary with the possibility the randomly selected session being an outlier, it nonetheless gives valuable insight on the discriminative power of the sentence embeddings. It should be noted that we do not make any assumptions on the meaning behind the clusters in this work other than their adeptness in classifying behavior.

Embeddings as features in supervised learning

We also evaluated two supervised techniques on both the IEMOCAP and Couples Therapy Corpus. The two methods are k-nearest neighbor and a more advanced neural network-based method, both of which utilize the unsupervised embeddings as features in supervised learning.

Datasets

In this section we describe the datasets that were used in the experiments. We used the OpenSubtitles2016 corpus (Lison & Tiedemann, 2016) to pre-train sentence embeddings in the online multitask framework. To evaluate the embeddings in domain-specific tasks, we used the Couples Therapy Corpus (Christensen et al., 2004) and IEMOCAP (Busso et al., 2008)¹.

Model architectures and training details

Results on couples therapy corpus

We defined two sub-tasks in behavior annotation on the Couples Therapy Corpus: (1) binary classification of the presences of behaviors and (2) regression for real-valued session ratings of the behaviors.

For the classification sub-task we used the accuracy averaged across all test folds as the evaluation metric. Table 2 shows the accuracy results on different behaviors in the Couples Therapy Corpus. The addition of the multitask objective improved the classification accuracy of sentence embeddings from the conversation model across all behaviors except Positivity in unsupervised classification with k-Means. Under supervised learning using k-NN, our multitask embeddings improved accuracy on all behaviors except Humor. In terms of mean accuracy, our multitask embeddings performed better than other sentence embeddings with an absolute improvement over no multitasking of 1.07% and 3.24% for unsupervised and supervised methods respectively. Our multitask embeddings also achieved the highest mean accuracy over all the behaviors. The improvement over the second best results obtained from GenSen was statistically significant with p-value <0.006 using McNemar’s test.

For the regression sub-task we evaluated performance using Krippendorff’s alpha coefficient (Hayes & Krippendorff, 2007). Krippendorff’s alpha is a reliability measure of the agreement between independent observers in regards to their annotation of data, commonly known as the inter-annotator agreement. We used this metric to evaluate how well trained models would function as a replacement for human annotators. Similar to Tseng, Baucom & Georgiou (2017) we evaluate the agreement with various ways of incorporating machine-generated ratings. In the first method, human annotations were randomly replaced by the estimated ratings in each session. This was performed 10 times to obtain the average Krippendorff’s alpha of random injection. In the second method, the outlier annotation (rating farthest from the mean) in each session was replaced by the estimated ratings.

Table 3 shows the inter-annotator agreement of the different injection methods. While no system was consistently optimal, we observed that our online MTL embeddings were comparable with state-of-the-art general purpose embeddings. In fact, statistical tests using Mann–Whitney U test on the annotation errors showed no significant differences between the best model and ours.

To factor out the influence of hyper-parameters and randomness of training we analyzed the performance of all the seq2seq models in our hyper-parameter search space. For each model configuration, five intermediate checkpoints from training were randomly selected. Sentence embeddings were then extracted from these individual models and applied to the behavior classification task. We then compared the performance of models with and without multitask learning. The standard error plot of the performance in Positivity and Negativity recognition is shown in Fig. 2. We observed that the addition of the multitask learning objective collectively increased performance in the final task for most behaviors. This shows that the addition of online transfer learning through multitask to unsupervised sentence embeddings does indeed provide an advantage in performance.

Results on IEMOCAP

We evaluated the performance of emotion recognition on IEMOCAP using weighted accuracy (WA) which avoids inflation due to imbalanced number of labels in each class. This is also equivalent to the macro-average of recall scores per class. In addition to general purpose embeddings we also compared with other works that only used IEMOCAP transcripts (Cho et al., 2018; Jin et al., 2015; Gamage, Sethu & Ambikairajah, 2017). It should be noted that there is no official consensus on train/test split or evaluation procedure in IEMOCAP, and while we made every effort to be consistent with past work (in terms of label classes, number of utterances used, and cross-validation scheme) the results may not be exactly comparable.

The results of emotion recognition on IEMOCAP are shown in Table 4. We observed that the addition of online MTL improved the accuracy of conversation model embeddings by an absolute value of 8.02%, which is more than 14% relative improvement. When comparing among our own implementations we observed that the highest accuracy was obtained using embeddings from the Universal Sentence Encoder which had a weighted accuracy of 64.83%. The system trained using our sentence embeddings offered a close second by less than one percent with 63.84% accuracy. Statistical analysis using McNemar’s test showed that the improvement of the best system over our proposed embeddings was not significant. However, we observed significant improvement from our model over embeddings from InferSent with p-value <0.02. Given the considerably smaller amount of pre-training data required and the simpler structure of our proposed MTL system this similarity in performance to Universal Sentence Encoder and advantage over other embeddings is notable.