Hierarchical transformer speech depression detection model research based on Dynamic window and Attention merge

Learnable speech split module

In previous works on audio feature processing, it is often difficult to determine the length of frame features for each word. It is also challenging to know the number of phonemes in each word. Therefore, using a fixed window size for feature processing is unreasonable. To address this issue, the current study utilizes textual data corresponding to speech and additional information such as timestamps for each word, to derive the dynamic window size S for each stage of feature processing and merge modules. The structure of the learnable speech split module (LSSM) module is depicted in Fig. 3.

After preprocessing the speech data, we can obtain timestamp information for each word and information about the sounds within each word. From this, we can determine the window size S_p for the phoneme stage features. S_p corresponds to the number of sounds within each word. Subsequently, the Word2Vec backbone network is employed to extract features from the sound information within each word, facilitating the transformation from text to vectors. Then, the textual features representing the phonemes, processed through the MLP network, are subjected to the Softmax function to obtain the phoneme weights. Finally, based on the weights of each phoneme within the word ω_i and the frame length of the word S_w, the frame length of each phoneme S_i is computed. This allows us to determine the window size S_f for frame-stage features. The specific formula for calculating the window size S_{f_i} of the ith feature window in the frame stage is as follows: (1)${\displaystyle S_{f\text{\_}i}=\left\{\begin{array}{cc}floor\left(S_i\times {\omega }_i\right),\hfill & i\in \left[1,n-1\right]\hfill \\ S_w-\sum _{j=1}^{n-1}{S}_{f\text{\_}j},\hfill & i=n\hfill \end{array}\right.}$

where S_i represents the adjusted feature window of the frame stage, ω_i represents the weight corresponding to the ith phoneme, and S_w represents the frame length of the word.

Adaptive attention merge

Each feature merge module facilitates better alignment between the various feature processing stages. However, if each feature merge module only uses average pooling and linear connections, it may blur feature details. This happens when merging the outputs of the previous feature processing stage. To address this issue, this article introduces the adaptive attention merge (AAM), which combines feature merge weights on top of average pooling. The structure of AAM is illustrated in Fig. 4.

Based on the dynamic window size S_f and S_w generated by the LSSM module, n features from the previous stage need to be fused into one feature for the next stage. However, this n is not fixed. Therefore, for features of different lengths (n) in different merge stages, this article designs different attention weight generation modules. The attention weight generation module comprises two fully connected layers, referred to as FC layers, to linearly transform the audio feature vectors to be merged. This module first elevates the n-dimensional features X to 2n dimensions through the first FC layer, the calculation principle of which is illustrated in Eqs. (2) and ((3). Using the same approach, the features undergo a second FC layer to reduce the dimensionality back to n dimensions. Subsequently, the features undergo mapping through a sigmoid activation function to map the linearly transformed feature vectors to the range between 0 and 1, obtaining initial feature merge weights, denoted as α_i. And the final feature merge weights ${\alpha }_i^{\prime }$ are obtained based on average pooling. The final feature merge weights ${\alpha }_i^{\prime }$ are used to compute the weighted sum of the audio feature vector X to obtain the merged feature vector X′. The specific formulas are as shown in Eqs. (2), (3), (4), (5): (2)${\displaystyle X\in R^{n\times c}=\left[X_1,X_2,\dots ,X_n\right],X_i\in R^{1\times c},i\in \left[1,n\right]}$ (3)${\displaystyle Y\in R^{2n\times c}=W^{T}X+B,W\in R^{n\times 2n},B\in R^{2n\times c}}$ (4)${\displaystyle {\alpha }_i^{\prime }=Softmax\left(\frac{1}{n}+{\alpha }_i\right),i\in \left[1,n\right]}$ (5)${\displaystyle X^{\prime }\in R^{1\times c}={\alpha }_1^{\prime }{X}_1+{\alpha }_2^{\prime }{X}_2+\cdots +{\alpha }_n^{\prime }{X}_n}$

where Y represents the features elevated to 2n dimensions, W denotes the weight matrix, and B represents the bias matrix.

Variable-length residual module

In response to the problem of differing feature lengths and the loss of original feature details resulting from the merging of features at each stage, this article proposes Variable-Length Residual Module (VL-RM). The module consists of two layers. The first layer is a variable-length average pooling layer. Since the lengths of the original features and the outputs of the merge modules between each stage are different, it is necessary to shorten the length of the original features based on the length of the features outputted by the merge module of the current stage, using a variable-length average pooling layer. The variable-length average pooling layer is depicted in Algorithm 2 . The second layer involves adding the original features processed by the first layer with the output features of the feature merging module and performing layer normalization.

 
_______________________________________________________________________________________________________ 
  Algorithm 2: Variable-Length Average Pooling Layer                    _________ 
    input  : Initial audio feature Xinitial with length linitial, and the 
                 feature length l’ of meraged audio feature X’. 
    output: Residual audio feature Xr. 
  1  pad_length ← 0; 
  2  if linitial mod l′ ⁄= 0 then 
     3   pad_length ← l′− (linitial mod l′); 
4   Xinitial ← Add pad_length pad to the end of Xinitial; 
  5  end if 
 6  X′initial ← Group Xinitial by (linitial+pad_length 
          l′      ); 
  7  Xr ← Apply AvgPool inside every groups in X′initial;    

Depression databases and evaluation metrics

The Distress Analysis Interview Corpus-Wizard of Oz (DAIC-WOZ) database (Gratch et al., 2014) is a real clinical dataset used in the Audio/Visual Emotion Challenge and Workshop (Ringeval et al., 2019) for depression recognition. The DAIC-WOZ is a subset of the Distress Analysis Interview Corpus (DAIC), renowned for its completeness, recognition, and high utilization among publicly available depression datasets. Through the interactive form of interview communication between the electronically manipulated virtual interviewer Ellie and the interviewees, audio-video data with a conversation duration exceeding 50 h was collected.The length of each audio file ranges from 7 to 33 min, with a fixed sampling frequency of 16,000 Hz. Furthermore, this dataset utilizes the Patient Health Questionnaire-8 (PHQ-8) scores from interviewees, along with a binary status label (PHQ-8 > 10 and PHQ-8 ≤ 10), as indicators of the interviewee’s mental health status. Additionally, the training and validation sets of DAIC-WOZ comprise audio data from 107 and 35 interviewees, respectively. Each speech segment in the DAIC-WOZ corpus contains the voices of both the interviewees and the interviewers, along with some irrelevant ambient noise, making it unsuitable for direct use in depression identification. Therefore, to address this issue, it is necessary to first isolate each question and answer between the interviewees and interviewers based on the timestamps in the text transcripts of the corpus, while simultaneously discarding irrelevant ambient noise. Since the DAIC-WOZ suffers from sample imbalance (100 healthy and 42 depressed interviewees) and some participants’ utterances are quite short, we resample the corpus in our experiment. In detail, when the participant is depressed, we select his/her longest 46 utterances as the training or testing samples. Otherwise, the longest 18 utterances are selected. By following the dataset split of DAIC-WOZ, we have 1,386 healthy utterances and 1,380 depressed utterances for training, while 414 healthy utterances and 552 depressed utterances for validation.

Currently, most evaluation metrics adopt accuracy, precision, recall, and F1 score. To better assess the performance of our model, and based on previous experimental experience, we have adopted three evaluation metrics: weighted accuracy (WA), unweighted accuracy (UA), and macro-average F1 score (MF1). The specific calculation are as shown in Eqs. (6), (7), (8), (9): (6)${\displaystyle WA=\frac{1}{\sum _{i=1}^N{W}_i}\sum _{i=1}^N{W}_i\times Accuracy\left(i\right)}$ (7)${\displaystyle UA=\frac{1}{N}\sum _{i=1}^{N}Accuracy\left(i\right)}$ (8)${\displaystyle F1=\frac{2\times Precision\times Recall}{Precision+Recall}}$ (9)${\displaystyle MF1=\frac{1}{N}\sum _{i=1}^{N}F1\left(i\right)}$

where W_i represents the number of samples in class i, Accuracy(i) and F1(i) represent the classification accuracy and F1 score for class i, respectively.

Experimental details

Comparative experiment

To demonstrate the effectiveness of the DWAM-Former model. On one hand, this article compares the DWAM-Former model with representative models published in recent years with the DAIC-WOZ dataset. On the other hand, as different feature extraction methods can affect the model’s detection performance, this article compares various feature extraction methods to better validate the generality and scalability of the DWAM-Former model.

Compared to the widely used deep learning techniques, DWAM-Former demonstrates superior performance on the DAIC-WOZ dataset, with MF1, UA, and WA reaching 0.788, 0.876, and 0.852 respectively. Detailed experimental results comparing different models are presented in Table 1. Compared to CNN-based models (FVTC-CNN, EmoAudioNet, Saidi, and TOAT), DWAM-Former exhibits improvements in MF1, UA, and WA by 15.88%, 20%, and 15.92% respectively. Moreover, compared to Solieman, which combines CNN and LSTM models, DWAM-Former demonstrates enhancements in MF1, UA, and WA by 29.18%, 32.68%, and 29.09% respectively. In contrast, when comparing the STFN model, which combines dilated convolution and LSTM, with the DWAM-Former model, there is an improvement of 7.95% in MF1 value and 5.97% in UA. When compared to the Transformer-based SIMSIAM-S model, DWAM-Former exhibits a 21.19% improvement in WA. Furthermore, when compared to hierarchical transformer models considering speech structure (SpeechFormer and SpeechFormer++), DWAM-Former demonstrates an increase of 7.5%, 8.22%, and 10.51% in MF1, UA, and WA respectively. The above results all highlight the detection accuracy of the DWAM-Former model.

Different hand-crafted speech features and pre-trained speech features based on self-supervised learning exhibit varying detection performance within the same model. The experimental data of different speech features on the DWAM-Former model and the comparison results with other models are presented in Table 1. The H/C speech features, Spec and Logmel, achieve MF1 values of 0.592 and 0.676 respectively on the DWAM-Former model. Compared to the SpeechFormer and STFN models, the MF1 values are improved by 6.09% and 7.81% respectively. Additionally, FBANK achieves an MF1 value of 0.757 on the DWAM-Former model, showing a 3.72% improvement compared to SpeechFormer++. The self-supervised pre-trained speech feature, Wav2Vec, achieves an MF1 value of 0.754 on the DWAM-Former model. Compared to SpeechFormer, STFN, and SIDD, it shows an improvement of 8.65% in MF1 value. Additionally, Wav2Vec 2.0 attains an MF1 value of 0.769 on the DWAM-Former model. Compared to TOAT and STFN, it exhibits an improvement of 5.34% in MF1 value. Furthermore, HuBERT achieves an MF1 value of 0.788 and a WA of 0.852 on the DWAM-Former model. Compared to SIMSIAM-S, WA is improved by 21.19%, and compared to SpeechFormer++, the MF1 value is enhanced by 11.14%. The aforementioned results collectively demonstrate the robust generalizability and scalability of the DWAM-Former model.

Ablation studies

Comparison of different feature merge methods

As speech features undergo multi-stage feature processing from fine to coarse, it is necessary to merge multiple features from the previous stage into one feature for the next stage. However, the importance of each feature from the previous stage for this single feature in the next stage varies. Therefore, this section compares different merge methods to validate the rationale behind the merge method used in this article. Specific experimental results are shown in Table 4.

Table 4:

Effect of different merging methods on model performance.

Merging method	WA	UA	MF 1
Average pooling	0.723	0.714	0.702
Attention Weight-D	0.729	0.721	0.707
Attention Weight-F	0.764	0.771	0.720
Average Pooling + (Attention Weight–D)	0.806	0.797	0.769
Average Pooling + (Attention Weight–F)	0.852	0.816	0.788

DOI: 10.7717/peerjcs.2348/table-4

Notes:

D denotes that the feature weights come from different dimensions of the feature, while F indicates that the feature weights come from different features within the same dimension.

The experimental results indicate that using only average pooling for feature merging yields the lowest performance in model detection, with MF1 values, UA, and WA being 0.702, 0.714, and 0.723, respectively. However, employing a learnable method, which assigns different attention weights to the features to be merged, results in improvements, with MF1 values, UA, and WA increasing by 2.56%, 7.98%, and 5.67%, respectively. Utilizing a learnable method for feature merging may lead the model to overly focus on certain features, thereby disregarding the importance of other features. Thus, this study combines the average pooling method with the attention weights obtained through a learnable approach to derive new feature weights. Ultimately, this results in a further improvement in model performance, with the MF1 value, UA, and WA increasing by 9.44%, 5.84%, and 11.52%, respectively.

Comparing the two different methods of obtaining feature weights, D and F, respectively determine the merge weights based on different dimensions of features and different features within the same dimension. The experimental results indicate that the feature weights obtained by method F lead to better detection performance compared to method D. Additionally, due to the variability in the number of features compared to the fixed number of feature dimensions, merging different numbers of features requires different merging networks, resulting in higher computational complexity for method F compared to method D. While method D offers a simpler network structure, its channel attention-like approach lacks robustness in merging different features compared to method F.

To further comprehend the model and ascertain the reasons behind the observed improvements, we analyze an utterance sample from DAIC-WOZ and compare the attention weights with and without AAM by visualization. The attention weights highlight the significance of each token within the model, determined by summing all the weights of the same value vector in MSA. In this utterance, there exists forced breathing which is an indicator of depression. As shown in Fig. 5, attention weights around the forced breathing become increasingly prominent with the help of the AAM module. However, without the AAM module, the higher attention weights in the first stage of the model are assimilated by the average pooling merge, stage by stage. This comparison illustrates the ability to avoid feature losses during merging operations for the AMM module.