Dual-stream transformer approach for pain assessment using visual-physiological data modeling

AI4PAIN

The AI4PAIN dataset (Fernandez-Rojas et al., 2024) is a comprehensive resource designed to support research in automatic pain assessment. Collected at the Human-Machine Interface Laboratory at the University of Canberra, Australia, this dataset includes multimodal data from participants subjected to controlled pain stimuli using a transcutaneous electrical nerve stimulation (TENS) device. The dataset captures facial video data at a 30Hz sample rate, recorded using a Logitech StreamCam, and physiological data from a functional near-infrared spectroscopy (fNIRS) headset placed on the frontal region of participants’ heads. Participants experienced varying levels of electrode pain, classified as low pain and high pain, with stimuli applied to different anatomical locations on the arm and hand. Each stimulus was repeated 12 times, providing robust data for both conditions. The dataset also includes a baseline period recorded at the start of each experiment, representing the No Pain condition.

The AI4Pain dataset is particularly valuable for developing and evaluating machine learning models aimed at automatic electrode pain detection and assessment, offering a rich source of data for studying the relationship between facial expressions, physiological responses, and perceived pain. According to data creators, a 60-s baseline was first recorded (B), which can serve as the No Pain condition. Additionally, the rest (R) period could be utilized to balance the dataset, as there are more samples of pain data (Low and High) than of no pain (baseline). The fNIRS includes 48 channels in various series lengths for each stimulus (sample shown in Fig. 2).

We selected a fixed number of samples from fNIRS data to synchronize with the facial video modality. The statistics of the dataset from First Multimodal Sensing Grand Challenge for Next-Gen Pain Assessment (AI4Pain) as shown in Table 2, each video lasts 10 s at 30 fps. For brain signal data, the sampling frequency was 10 Hz, and no pre-processing was applied, as it was expected that deep learning models could learn the neural representation of pain from raw fNIRS data. For learning, we set the window length for each video sample at 16 frames with a temporal sample rate of 48, and the fNIRS window time size is 200. Note that we treat the public validation set from the challenge as our test set.

BioVid heat pain

The BioVid Heat Pain Database (Walter et al., 2013) is a specialized dataset designed for automatic pain monitoring systems research, focusing on how facial expressions vary with different intensities of heat-induced pain. It provides high-quality labels corresponding to varying pain stimuli, enabling detailed analysis of facial activity in response to pain. The BioVid Database part B is designed for pain intensity recognition, containing 8,600 samples collected from 86 subjects. The data includes both frontal video recordings and 5-dimensional biomedical signals, which consist of GSR, ECG, and EMG at the trapezius, corrugator, and zygomaticus muscles. These signals are available in both raw and preprocessed formats. On the other hand, part A includes 8,700 samples from 87 subjects, but only GSR, ECG, and EMG at the trapezius are measured.

The dataset is generally organized into five pain intensity classes, with 20 samples per class for each subject, and each sample spans a time window of 5.5 s with recognized pain intensity. Each video sample is segmented into windows of 16 frames for training, with a temporal sampling rate of 16. For multivariate signal data, the window size is set to 400. Regarding biosignal feature extraction, GSR, EMG, and ECG at a sampling rate of 512 Hz are processed (as shown in Fig. 3) as follows: GSR signals are processed without filtering and standardized before feature extraction; EMG is first filtered with a Butterworth bandpass filter (20–250 Hz) to isolate the bursts that carry the information about the muscle activity. The signal is subsequently refined and denoised through an Empirical Mode Decomposition (EMD)-based method de Oliveira Andrade (2005). ECG is filtered with a Butterworth bandpass filter (0.1–250 Hz) to retain key cardiac activity while removing baseline wander and high-frequency noise.

Table 2 includes the statistical distribution of the BioVid-B dataset in our experiment, which we split into 69 subjects for training and 17 subjects for testing. In our rich suite of experiments, we prefer part B over part A, which has extra sensing signals for multivariate time series learning. To ensure clearer class separation and align the setup with the binary structure of the AI4Pain dataset, we selected PA2 as “low pain” and PA4 as “high pain.” This split is due to make corpora similar to the AI4Pain dataset, and to the best of our knowledge, no prior work has been conducted on this specific part of the dataset. In addition, we made umap analysis across samples in BioVid dataset (different in gender, age, country,…), and the signals are standardized (Fig. 4).

Facial pain representation learning

We aim to create pre-training for domain-specific pain assessment. Video data contains temporal relations, unlike static images. Thus, our video-based MAE (Fig. 6A) focuses on reconstructing the intricate spatio-temporal characteristics observed in facial videos to create a robust and transferable facial representation from unseen labeled data.

Face model. We begin with creating n masked and $(k-n)$ visible tokens (k is the total number of tokens obtained from random video temporal sampling ${\mathbf{x}}_{\mathbf{t}}$). The masking ratio can be pre-defined as $r=\frac{n}{k}$. The comprehensive MAE for video v modality as an asymmetric design consists of an encoder ${\mathcal{E}}_v$ to operate only on visible partial and map to lower facial representation space, a decoder ${\mathcal{D}}_v$ that maps the latent space to the reconstructed full signals, and a discriminator $f_v$ for enhancing synthesis through adversarial training.

(1) $${\hat{\mathbf{x}}}_m={\mathcal{D}}_v\circ {\mathcal{E}}_v({\mathbf{x}}_v)$$

The encoder ${\mathcal{E}}_v$ first embeds the input visible patches ${\mathbf{x}}_v\in {\mathbb{R}}^{(k-n)\times d_{model}}$. Here, $\circ$ denotes composition of relations between ${\mathcal{E}}_v$ and ${\mathcal{D}}_v$. Accordingly, ${\mathcal{D}}_v$ reconstructs to n masked tokens. We further integrate adversarial adaptation into our MAE backbone to enhance generation quality, leading to more robust latent features by deploying $f_v$ discriminator as an multilayer perceptron (MLP)-based network.

The attention map for the temporal dimension $A_{temporal}^{(h,w)}$ is computed as follows:

(2) $$A_{temporal}^{(h,w)}=softmax\left(\frac{Q^{(h,w)}\cdot {(K^{(h,w)})}^T}{\sqrt{d_k}}\right)\cdot V^{(h,w)}$$where $Q^{(h,w)}$, $K^{(h,w)}$, $V^{(h,w)}$ are the query, key and value matrices at spatial location $(h,w)$ and $d_k$ are the dimensions of the query/key and value vectors.

Self-supervised training. Our facial video MAE model is optimized by $L_2$ reconstruction loss with given an input masked tokens ${\mathbf{x}}_m$, the MAE module reconstruct it back to ${\hat{\mathbf{x}}}_m$:

(3) $${\mathcal{L}}_{\mathrm{r}\mathrm{e}\mathrm{c}}=\frac{\mathbf{1}}{\mathbf{N}}\sum _{\mathbf{i}=\mathbf{1}}^{\mathbf{N}}\Vert {\mathbf{x}}_{\mathbf{m}}^{(\mathbf{i})}-{\hat{\mathbf{x}}}_{\mathbf{m}}^{(\mathbf{i})}{\Vert }_{\mathbf{2}}$$where N is the total number of data points, ${\mathbf{x}}^{(i)}$ and ${\hat{\mathbf{x}}}^{(i)}$ denote the masked token reconstructions for the i-th data point. The adversarial adaptation takes the Wasserstein GAN loss (Arjovsky, Chintala & Bottou, 2017) as follows:

(4) $$\begin{array}{rl}{\mathcal{L}}_{\mathrm{a}\mathrm{d}\mathrm{v}}& ={\mathbb{E}}_{{\mathbf{x}}_m\sim {\mathbb{P}}_r}[f({\mathbf{x}}_m)]-{\mathbb{E}}_{z_v}\sim {\mathbb{P}}_{z_v}[f({\mathcal{D}}_v(z_v))]\\ & +\lambda {\mathbb{E}}_{\hat{\mathbf{x}}\dim {\mathbb{P}}_{\hat{\mathbf{x}}}}{\left(||{\mathrm{\nabla }}_{{\hat{\mathbf{x}}}_m}f({\hat{\mathbf{x}}}_m)|{|}_2-1\right)}^2\end{array}$$where $\mathbf{x}\sim {\mathbb{P}}_r$ is the real sample from the true data, $\mathbf{z}\sim {\mathbb{P}}_z$ denotes latent feature $z_v$ from latent space distribution and $\lambda$ is a hyperparameter that controls the strength of the gradient penalty.

Multivariate time-series compression

We proposed MAE model for multivariate biosignal representations learning, which is capable of reconstructing incomplete and irregular time-series data by learning robust representations that capture temporal dependencies and correlations across diverse biosignal modalities.

Time series masking. We first ensure that each feature of the time series is normalized independently to the range $[-1,1]$, as shown in Eq. (5):

(5) $${\mathbf{X}}_{\mathbf{i}\mathbf{j}}^{\prime }=\mathbf{2}\cdot \frac{{\mathbf{X}}_{\mathbf{i}\mathbf{j}}-{\mathbf{X}}_{\mathbf{j},\mathrm{m}\mathrm{i}\mathrm{n}}}{{\mathbf{X}}_{\mathbf{j},\mathrm{m}\mathrm{a}\mathrm{x}}-{\mathbf{X}}_{\mathbf{j},\mathrm{m}\mathrm{i}\mathrm{n}}}-\mathbf{1}$$

Time series mask generation. We treat all types of biomedical signals uniformly and do not apply any specific preprocessing. To generate a time-series mask, we first randomly select a continuous segment of the input sequence with C channels and T time steps $\mathbf{X}\in {\mathbb{R}}^{C\times T}$ of length $\mathbf{L}\le T$. For each channel, we create a binary temporal mask ${\mathbf{m}}_{\mathbf{t}}\in \{0,1{\}}^L$ by simulating a two-state Markov process with states “keep” (1) and “mask” (0). The transition probabilities are determined by the target masking ratio $\mathbf{r}$ and average masked segment length ${\mathbf{l}}_{\mathbf{m}}$, computed as:

(6) $$p_m=\frac{1}{{\mathbf{l}}_{\mathbf{m}}},p_u=\frac{p_m\cdot \mathbf{r}}{1-\mathbf{r}},$$where $p_m$ is the probability of switching from “mask” to “keep”, and $p_u$ is the probability of switching from “keep” to “mask”. The process iterates sequentially over the $\mathbf{L}$ time steps, independently for each channel, producing the final mask ${\mathbf{m}}_{\mathbf{t}}$.

Time series model. We consider physiological sensing biosignals as multivariate time series data. Our model, as described in Fig. 6B, builds upon autoencoder architecture with encoder ${\mathcal{E}}_{ts}$ and decoder ${\mathcal{D}}_{ts}$. Given a query and a set of key-value pairs are used to reconstructed by ( ${\mathcal{E}}_{ts}$, ${\mathcal{D}}_{ts}$). Specifically, for an input sequence ${\mathbf{x}}_{\mathbf{t}}=\left\{x_1,x_2,\dots ,x_L\right\}$ and padding masks ${\mathbf{m}}_{\mathbf{t}}\in \{0,1{\}}^m$:

(7) $$\begin{array}{c}{\mathbf{m}}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}={\mathbf{x}}_{\mathbf{t}}\odot {\mathbf{m}}_{\mathbf{t}}\\ {\hat{\mathbf{x}}}_{\mathbf{t}}={\mathcal{D}}_{ts}\circ {\mathcal{E}}_{ts}({\mathbf{m}}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}})\\ {\hat{x}}_j={\hat{\mathbf{x}}}_{\mathbf{t}}[j]\mathrm{f}\mathrm{o}\mathrm{r}j=1,2,\dots ,L\end{array}$$

We train the self-supervised pre-training with $L_2$ reconstruction loss similar to Eq. (3) between ${\mathbf{x}}_{\mathbf{t}}$ and ${\hat{\mathbf{x}}}_{\mathbf{t}}$ resulted by ${\mathcal{D}}_{ts}$.

Attention fusion classifier

We employ an attention-based fusion classifier to learn from facial video and biomedical signals. To avoid overfitting-prone MLPs, which could diminish the generalizability of the model and lead to less accurate predictions, we use lightweight attention embeddings for effective alignment and integration (Fig. 5), optimizing with cross-entropy loss.

The output tokens from the MAE encoders ( $\mathcal{E}v,\mathcal{E}ts$) in Fig. 7 Green form multivariate time series features with varying lengths and dimensions. We standardize them using absolute positional encoding (tAPE) (Foumani et al., 2024) and efficient Relative positional encoding (eRPE) (Foumani et al., 2024). tAPE embeds sequence position based on input dimensions and length ( $d_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}=16$), helping greatly reduce model size while preserving temporal structure. eRPE further enhances self-attention via learnable scalar weights $w_{i\text{-}j}$ (i.e., $w\in {\mathbb{R}}^{O(L)}$) which represents the relative position weight between positions $i$ and $j$.

We apply tAPE before input embedding and perform attention with eRPE inside a multi-head attention block, followed by a feedforward MLP. This hybrid positional attention (HPA) mechanism is illustrated in Fig. 7 (purple).

We adopt the dense co-attention symmetric network (DCAN) in our prediction module (Fig. 7: yellow) to enable balanced, bidirectional interaction between facial and physiological features. Originally developed for visual question answering (Nguyen & Okatani, 2018) and later adapted for multimodal emotion recognition (Zhao, Liu & Lu, 2021), DCAN facilitates effective multimodal fusion beyond standard query-driven attention. The complete feature embedding and fusion procedure is outlined in Algorithm 1.

Model settings

We implemented our model using PyTorch. For facial video in pain domain-specific settings, our MAE-3DViT (22.5 M) is optimized for more lightweight than ViT-based (86 M) and ViT-Large (307 M).

We empirically implement 3D-ViT Small (22.5 M), which is a more practical choice despite a slight trade-off in accuracy in Table 3 (77.66% compared to 77.85% ViT-large on Ai4Pain dataset), as it is easier to deploy in real-world applications, whereas ViT Large demands significantly more computational resources and hardware. Hence, the model consists of 12 transformer blocks in ${\mathcal{E}}_v$ and eight transformer blocks in ${\mathcal{D}}_v$ with six attention heads and an embedding size of 384. Video MAE takes the input of image size 224 $\times$ 224 pixels, a patch size is 16 in 16 frames with a high mask proportion of 0.9.

For the bio-psychology modality, both ${\mathcal{E}}_{ts}$ and ${\mathcal{D}}_{ts}$ utilized Transformer encoders as our structure omits the Transformer decoder component, as the requirement for (masked) ground truth output sequences makes it unsuitable for certain downstream tasks, particularly classification. Moreover, the Transformer encoder is extensively utilized for generative tasks, particularly when the output sequence length is not pre-specified. This is exemplified in applications such as translation and summarization in natural language processing and forecasting in time series analysis. Each of ${\mathcal{E}}_{ts}$ and ${\mathcal{D}}_{ts}$ has three transformer encoder layers with four heads of attention, the model dimension is set to 64, and the maximum mask ratio is set to 0.3.

In the lightweight classifier setting, HPA has an embedding size of 16 and the dense feedforward part of the transformer layer has a dimension of 256 with eight attention heads. Lastly, the fusion DCAN includes two dense co-attention (DCA) layers. The average total of params for the whole multimodal is 24.5 M between the two dataset settings (as shown in Table 4).

Training

We performed all the training on dual Nvidia RTX 8000 GPUs. In the first stage of our dual MAE pre-training, we train solely for ${\mathcal{E}}_v$ and ${\mathcal{E}}_{ts}$. In particular, ( ${\mathcal{E}}_v,{\mathcal{D}}_v$) is trained with 20% set of BioVid-A for proper performance ViT structure. In terms of setting params, facial compression training used the AdamW optimizer with a base learning rate of $1.5e-4$, momentum parameters ${\beta }_1=0.9$ and ${\beta }_3=0.95$, and a cosine decay learning rate scheduler, the masking ratio is set to 0.9. For ( ${\mathcal{E}}_{ts},{\mathcal{D}}_{ts}$), we employed the Adam optimizer with ${\beta }_1=0.5$, ${\beta }_2=0.9$, a base learning rate of $1e-4$. ${\mathcal{D}}_{ts}$ takes the learning rate of $1e-4$ with RAdam optimizer and a batch size of 64. In the second stage, we trained the full classifier in Fig. 5B with frozen ${\mathcal{D}}_v$ and fine-tuned ${\mathcal{D}}_{ts}$ heads until converged. We applied a learning rate of $1e-3$ with RAdam optimizer. To prevent overfitting and improve stability, early stopping is implemented with a patience of 10 epochs. Additionally, a ModelCheckpoint callback is set up to save the best model weights based on the validation loss.

Baselines

We conduct a comprehensive comparison against well-established handcrafted baselines to ensure a detailed evaluation of our approach, which includes the following modalities:

Video. We adopt several recognized deep learning architectures as baseline models for the video modality. Specifically, we employ ResNet50 (He et al., 2016), and 3DViT (Alexey, 2020) backbones for temporal cube input. Furthermore, our comparison encompasses results from previous works (Gkikas & Tsiknakis, 2024; Prajod et al., 2024) and (Fernandez Rojas et al., 2023), all conducted on the AI4Pain dataset. We also reimplement the methodology from Bargshady et al. (2019).

Time-series. For the time-series modality, we utilized LSTM (Hochreiter & Schmidhuber, 1997) networks and the vanilla Transformer (Vaswani et al., 2017) augmented with additional convolutions to handle multidimensional data.

Multimodal. The existing literature on multimodal approaches that combine video and biomedical signals for pain assessment is limited. Consequently, we adapt techniques from visual-audio emotion recognition, as detailed in Tzirakis et al. (2017) and Dai et al. (2020) to establish our multimodal baselines, transforming 1D audio data into 2D time-series.

Evaluation method

To evaluate our model’s ability to classify pain condition labels, as defined by the dataset creators, we employ an extensive set of quantitative metrics: precision, recall, F1-score, and accuracy. Each metric provides a unique perspective on the model’s effectiveness, particularly in the context of classification tasks. In our experiments, we clarify that our results were obtained by extracting facial characteristics from ${\mathcal{E}}_v$, which was trained on only 20% of the BioVid-A data set (BioVid-B presents a disadvantage due to the placement of the EMG device on the forehead), which has been consistently applied in both datasets. We conducted on Biovid dataset part B for a comprehensive three pain levels evaluation and part A for binary and multi-class (five levels) classifications against SOTA by leave-one-subject-out (LOSO) cross-validation.

Component-wise evaluation

We conducted ablation studies to assess the impact of the cross-attention fusion module and the HPA embedding module on model performance. To evaluate the contribution of each module, we performed experiments using the proposed method with and without the respective components. The results, summarized in Tables 5 and 6, highlight the significance of these modules. In the two test cases, where the pain-level embedding module was excluded, including the HPA module led to improvements across all pain classes. Specifically, the video unimodal approach with the HPA module surpasses vanilla late fusion, while the signal model achieves a fair margin in accuracy. The DCAN fusion slightly improved overall evaluation metrics, benefiting from the cross-modality approach. In summary, the full architecture network achieved the highest overall evaluation for each pain class dataset.