CIT-EmotionNet: convolution interactive transformer network for EEG emotion recognition

CNN-based methods

In recent years, convolutional neural network (CNN)-based approaches (Kwon, Shin & Kim, 2018; Chen et al., 2019; Shen et al., 2019; Li et al., 2022a; Salama et al., 2018; Shen et al., 2022) are known for their capabilities of efficient feature extraction and powerful feature representation. Hence, many researchers have explored using CNN for extracting features from EEG signals for the emotional recognition of electroencephalogram (EEG) signals. Kwon, Shin & Kim (2018) used CNN for feature extraction from EEG signals. In this model, the EEG signal is preprocessed before convolution by wavelet transform while both the time and frequency of the EEG signal are taken into account (Kwon, Shin & Kim, 2018). Yang, Han & Min (2019) presented a multi-column CNN-based model for emotion recognition from EEG signals. The multi-column CNN-based model, whose decision was produced by a weighted sum of the decisions from individual recognizing modules (Yang, Han & Min, 2019). Based on the time and frequency combination EEG features, Chen et al. (2019) proposed a deep CNN model similar to that used for image classification in computer vision. The model not only avoided the labor-intensive process of manual feature extraction and feature selection prior to traditional machine learning classification but also effectively improved the accuracy and stability of EEG emotion recognition (Chen et al., 2019). Liu et al. (2020) came up with a kind of dynamic differential entropy (DDE) algorithm to extract the features of EEG signals. Subsequently, the extracted DDE features were classified by using a CNN (Liu et al., 2020). Rahman et al. (2021) suggested transforming EEG signals into brain topographic mapping covering frequency and spatial information, and classifying emotions by using CNN. Gilakjani & Al Osman (2022) proposed a deep learning model that combines CNN and residual recurrent neural networks (RNN) to extract both spatial and temporal features from multi-channel EEG signals. Not only 2D CNN have achieved good results in terms of EEG emotion recognition tasks, but 3D CNN has done remarkable work. For instance, Salama et al. (2018) investigated the application of 3D convolutional neural networks (3D-CNN) in emotion recognition and developed a data enhancement stage to improve the performance of 3D CNN models. Cho & Hwang (2020) presented a 3D CNN model that can efficiently represent the spatial-temporal representation of EEG signals for emotion recognition. Although CNN has achieved remarkable success in recognizing emotions from EEG signals and possesses advantages in local feature extraction, it is a locally sensitive model where each convolutional kernel only focuses on local features. This tends to result in the neglect of global features during feature extraction, making it difficult to capture the sequential information in EEG signals. This limitation can affect its performance in EEG emotion recognition tasks (Zhao et al., 2023).

Transformer for EEG classification

Transformer based on global self-attention is gaining increasing attention from researchers. The Transformer was first proposed to apply in the field of natural language processing, then it was introduced to the field of computer vision (Han et al., 2023). Vision Transformer (ViT), a complete self-attention-based image recognition model, has obtained state-of-the-art results in object detection (Hong et al., 2022), image classification (Chen, Fan & Panda, 2021), and segmentation (Gu et al., 2022). There were some studies that apply the Transformer for EEG signals classification. The studies showed that the attention mechanism in a Transformer can capture global information of EEG signal sequences and extract more relevant features for classification (Wang et al., 2023). For epileptic seizure prediction based on EEG signals, Hussein, Lee & Ward (2022) introduced a Transformer-based approach named Multi-Channel Vision Transformer (MViT) to automatically and simultaneously learn spatio-temporal-spectral features in multichannel EEG data. For the task of emotion recognition based on EEG signals, Liu et al. (2022) proposed an EEG emotion Transformer (EeT) model that learns spatial-spectral features directly from EEG signal sequences, thus making the traditional Transformer model suitable for EEG signals. Wang et al. (2023) introduced a model called Joint-Dimension-Aware Transformer (JDAT) for EEG emotion recognition, which was based on the squeezed multi-head self-attention (MSA) mechanism. By applying adaptive squeezed MSA to multidimensional features, JDAT was able to focus on diverse EEG information, including space, frequency, and time (Wang et al., 2023). Wang et al. (2022b) proposed a Transformer-based model for hierarchical learning of differentiating spatial information from the electrode level to the brain region level. Although the above-mentioned models have shown the effectiveness of Transformer models in EEG classification tasks, it should be noted that Transformer models are not particularly adept at handling local features (Zhao et al., 2023). Therefore, using only a Transformer model may overlook spatial information in EEG signals.

CNN and transformer fusion

To draw on the advantages of the CNN model featuring local information extraction and the Transformer model characterized by global information extraction, researchers (Zhang, Liu & Hu, 2021; Jiang et al., 2022; Shen et al., 2023b; Jiang et al., 2022; Song, Liu & Ma, 2022) have attempted to fuse these two models. In the field of visual recognition, Peng et al. (2021) introduced a novel network architecture called Conformer, which combined the strengths of convolutional operations and self-attention mechanisms to improve representation learning. Conformer was built upon the feature coupling unit (FCU), which interactively integrated local features and global feature representations at different resolutions (Peng et al., 2021). In the field of medical images, Zhang, Liu & Hu (2021) proposed a parallel TransFuse model to combine Transformer and CNN, in which both global dependency and low-level spatial details can be efficiently captured in a shallower way. In addition, a new fusion technique, BiFusion module, was proposed to efficiently fuse the multilevel features of both branches (Zhang, Liu & Hu, 2021). The TD-Net model proposed by Song, Liu & Ma (2022) combines Transformer and CNN to extract global and local information, using the former as an encoder to extract the global information and the latter to obtain the feature mapping of the image. The coding results from the Transformer are then connected to the CNN upsampling process (Song, Liu & Ma, 2022). Jiang et al. (2022) designed a new network model MTPA_Unet which combines Transformer and CNNs in a serial manner. In the field of remote sensing image semantic segmentation, Gao et al. (2021) proposed the STransFuse model that combines the advantages of both Transformer and CNN, which uses a staged model to extract coarse-grained and fine-grained feature representations at different semantic scales. Wang et al. (2022a) presented a Coupled CNN and Transformer Network (CCTNet) that combines the local details of the CNN and the global context of the Transformer. This model fuses the features of CNN and Transformer branches while maintaining the advantages of each. Additionally, the light adaptive fusion module (LAFM) and the coupled attention fusion module (CAFM) are also proposed to smoothly fuse the features of CNN and Transformer (Wang et al., 2022a). Zhang et al. (2022) put forward a model fusing Transformer and CNN, in which, the encoder module extracts features through Swin Transformer and the decoder module draws on some effective blocks from the CNN model. In the field of EEG-based recognition, Zhao et al. (2023) proposed an interactive approach for seizure detection that combines both local and global features. Local features are extracted using convolution operations, while global representations of the EEG signals are obtained through a self-attention mechanism. However, the previously mentioned model neglects the fusion of global and local features at each stage, resulting in a limited extraction of comprehensive feature information. Therefore, effectively integrating the global and local features of EEG signals at different stages to improve performance in emotion recognition remains a formidable challenge in the domain of EEG emotion recognition.

Overview

In this article, we utilize time-frequency feature extraction methods to derive the power spectral density (PSD) features across five frequency bands $\left\{\delta ,\theta ,\alpha ,\beta ,\gamma \right\}$ from all EEG channels in the EEG signal samples. The formulation of PSD is presented in Eq. (1).

(1) $$PSD=E[x^2],$$we define $S=(S_1,S_2,...,S_B)\in {\mathbb{R}}^{N_e\times B}$ as the frequency features containing B frequency bands extracted from EEG signals, where $N_e$ is the number of electrodes. Based on the electrode positions on the scalp, we construct the spatial-spectral representation $S^M=(S_1^M,S_2^M,...,S_B^M)\in {\mathbb{R}}^{H\times W\times B}$, where H and W denote the height and width of the frequency map respectively. $\left\{\delta ,\theta ,\alpha ,\beta ,\gamma \right\}$ is selected as the frequency band set.

The overview of the proposed CIT-EmotionNet model structure is depicted in Fig. 1. The raw EEG signals are first converted into EEG feature representations. Subsequently, these EEG feature representations are fed into CIT-EmotionNet for EEG emotion recognition. CIT-EmotionNet is primarily composed of three parts: CNN branch, Transformer branch, and CIT module. The EEG feature representations are input to the CNN branch and Transformer branch. The CNN branch consists of five stages, namely stage 0, stage 1, stage 2, stage 3, and stage 4. Each stage of the CNN branch from stage 1 to stage 4 contains two ResNet basic blocks, which respectively output local features C1, C2, C3, and C4. Similarly, the Transformer branch also consists of five stages, namely stage 0, stage 1, stage 2, stage 3, and stage 4. Each stage of the Transformer branch from stage 1 to stage 4 contains three Transformer Encoder blocks, which respectively output global features T1, T2, T3, and T4. The local features C1, C2, and C3, as well as the global features T1, T2, and T3, are input into the CIT module of stages 1, 2, and 3, respectively, where they are interactively fused. The CIT module comprises two blocks: the global to local block and the local to the global block. The output of the CIT module for each stage is the input for the next stage of both the CNN branch and the Transformer branch. Specifically, the output of the global to local block corresponds to the input for the next stage of the CNN branch, while the output of the local to global block corresponds to the input for the next stage of the Transformer branch. Finally, the local features C4 from stage 4 of the CNN branch and the global features T4 from stage 4 of the Transformer branch are merged through a Concat module, which is then fed into a fully connected layer for emotion classification recognition.

EEG feature representation

Figure 2 shows the process of converting the original EEG signals into spatial-spectral representations. Data were collected as previously described in Lu, Tan & Ma (2023). Specifically, we divide the original EEG signals into non-overlapping periods lasting for 4 s, and each segment is assigned the same label as the original EEG signals.

To construct the EEG feature representation, the temporal-frequency feature extraction method is used to extract the PSD features of five frequency bands $\left\{\delta ,\theta ,\alpha ,\beta ,\gamma \right\}$ of all EEG channels from the EEG signal samples in the EEG segments with a length of 4 s. We define $S=(S_1,S_2,...,S_B)\in {\mathbb{R}}^{N_e\times B}$ as a frequency feature containing a frequency band extracted from the PSD feature, in which the frequency band is $B\in \left\{\delta ,\theta ,\alpha ,\beta ,\gamma \right\}$, the electrode is $N_e\in \{FP1,FPZ,...,CB2\}$, and $X_b^B=(x_b^1,x_b^2,...,x_b^N)\in {\mathbb{R}}^N,(b\in \{1,2,...,B\})$ represent the collection of EEG signals from all $N_e$ electrodes on the frequency band B. Then, the selected data are mapped to a frequency domain brain electrode location matrix $S_b^M\in {\mathbb{R}}^{H\times W},(b\in \{1,2,...,B\})$ according to the electrode location on the brain. Here, the superscript M indicates that this feature is represented as a feature matrix. Finally, the frequency-domain brain electrode position matrices from different frequencies are superimposed to form the spatial-spectral feature representation of EEG signals, that is, the construction of the EEG feature representation $S^M=(S_1^M,S_2^M,...,S_B^M)\in {\mathbb{R}}^{H\times W\times B}$ is finished.

CNN branch

The Residual Network (ResNet) was proposed to address the problem of degradation in deep CNN models. ResNet utilizes residual connections to link different convolutional layers, thereby enabling the propagation of shallow feature information to the deeper layers. Thus, it is suitable to be used for local feature extraction from EEG feature representation. The specific structure of the CNN branch is shown in Fig. 3.

The input of ResNet is the spatial-spectral feature representation $S^M=(S_1^M,S_2^M,...,S_B^M)\in {\mathbb{R}}^{H\times W\times B}$. The spatial-spectral feature representation first goes through Stage 0, which consists of a $7\times 7$ convolutional layer, a max pooling operation, and batch normalization. The purpose of this stage is to perform initial convolutional feature extraction and downsampling of the EEG feature representation, reducing the size of the feature map while increasing the level of abstraction. Stage 0 preprocesses the input EEG feature representation through convolution, pooling, and batch normalization operations, transforming it into a smaller and more abstract feature representation, providing better input features for subsequent convolutional blocks. Specifically, the input of Stage 0 is the spatial-spectral feature representation $S^M=(S_1^M,S_2^M,...,S_B^M)\in {\mathbb{R}}^{H\times W\times B}$, where the shape of the spatial-spectral feature representation is $H\times W\times C$, with H representing the height, W representing the width, and C representing the number of channels. Since the number of frequency bands is $5$, $C=5$. However, this does not meet the input requirements of the original ResNet model since the first convolutional layer in the original ResNet model requires three input channels. If we directly use the original model to process data with five input channels, channel conversion or padding operations are required, which can be tedious. Therefore, we replaced the first half of the ResNet model with a new convolutional layer that has five input channels, $64$ output channels, a kernel size of $7\times 7$, a stride of $2$, a padding of $4$, and no bias. After passing through the new convolutional layer, the size and number of the output feature maps have changed. The equation for calculating the output shape after features pass through a convolutional layer is shown in Eq. (2).

(2) $$output\mathrm{\_}size=\frac{input\mathrm{\_}size+2\ast padding-kernel\mathrm{\_}size}{stride}+1.$$

By substituting the kernel size of $7\times 7$, stride of $2$, and padding of $4$ into the equation, the shape of the output features after the convolutional layer can be determined. After the convolutional layer, there is another max pooling layer. The calculation formula for the max pooling layer is as shown in Eq. (3).

(3) $$output\mathrm{\_}size=\frac{input\mathrm{\_}size-kernel\mathrm{\_}size}{stride}+1$$

By substituting the kernel size of $3\times 3$ and stride of $2$ into the equation, the shape of the output features after the max pooling layer can be determined. Due to the need to round down, the division in the formula is more accurately described as finding the quotient. Specifically, after passing through Stage 0, the shape of the output feature map is $\frac{H}{4}\times \frac{W}{4}\times 64$. This feature map will be used as the input for the subsequent Stage 1. The equations for Stage 0 of ResNet are shown in Eq. (4).

(4) $$C_0=\mathrm{M}\mathrm{a}\mathrm{x}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}(\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(\mathrm{B}\mathrm{N}(\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{7\times 7}(S^M)))),$$where $S^M$ is the input of Stage 0 in the CNN branch, and $C0$ is the output of Stage 0 in the CNN branch. $\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{7\times 7}(\cdot )$ represents the convolutional layer operation with an output channel of $64$, kernel size of $7\times 7$, the stride of $2$, and padding of $4$. $\mathrm{B}\mathrm{N}(\cdot )$ represents the batch normalization layer operation, which performs batch normalization on the output of the convolutional layer. $\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(\cdot )$ represents the ReLU activation function, which applies the ReLU activation function to the output of the batch normalization layer. $\mathrm{M}\mathrm{a}\mathrm{x}\mathrm{P}\mathrm{o}\mathrm{o}\mathrm{l}(\cdot )$ represents the max pooling layer operation, which performs max pooling using a $3\times 3$ pooling kernel, a stride of $2$, and padding of $1$.

The features output from Stage 0 are processed through Stage 1, Stage 2, Stage 3, and Stage 4 to generate the C1, C2, C3, and C4 features, respectively. After passing through Stage 1, the shape of the output feature C1 remains the same as the input feature, which is $\frac{H+8}{2}\times \frac{W+8}{2}\times 64$. After passing through Stage 2, the shape of the output feature C2 is $\frac{H+16}{4}\times \frac{W+16}{4}\times 128$. After passing through Stage 3, the shape of the output feature C3 is $\frac{H+32}{8}\times \frac{W+32}{8}\times 256$. After passing through Stage 4, the shape of the output feature C4 is $\frac{H+64}{16}\times \frac{W+64}{16}\times 512$. Each of Stage 1, Stage 2, Stage 3, and Stage 4 consists of two BasicBlocks. In BasicBlock, the input feature is added to the main branch output feature via a shortcut connection before being passed through a ReLU activation function. The equation of the main branch, as shown in Eq. (5).

(5) $$X_{main}=\mathrm{B}\mathrm{N}(\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{3\times 3}(\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(\mathrm{B}\mathrm{N}(\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{3\times 3}(X_{BasicIN}))))),$$where $X_{BasicIN}$ is the input of BasicBlock, $X_{main}$ is the output of the main branch in the BasicBlock. $\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{3\times 3}(\cdot )$ represents the convolutional layer operation with a kernel size of $3\times 3$, the stride of $1$, and padding of $1$. $\mathrm{B}\mathrm{N}(\cdot )$ represents the batch normalization layer operation, which performs batch normalization on the output of the convolutional layer. $\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(\cdot )$ represents the ReLU activation function, which applies the ReLU activation function to the output of the batch normalization layer.

The shortcut connection allows the gradient to flow directly through the network, bypassing the convolutional layers in the main branch, which helps to prevent the vanishing gradient problem. The equation of the shortcut connection, as shown in Eq. (6).

(6) $$X_{shortcut}==\mathrm{B}\mathrm{N}(\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}(\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{3\times 3}(X_{BasicIN}))),$$where $X_{BasicIN}$ is the input of BasicBlock, $X_{shortcut}$ is the output of the shortcut connection in the BasicBlock. $\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{3\times 3}(\cdot )$ represents the convolutional layer operation with a kernel size of $3\times 3$, the stride of $1$, and padding of $1$. $\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}(\cdot )$ represents the convolutional layer operation with a kernel size of $1\times 1$. $\mathrm{B}\mathrm{N}(\cdot )$ represents the batch normalization layer operation. $\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(\cdot )$ represents the ReLU activation function.

The addition of the input feature to the main branch output feature allows the network to learn residual mappings, which can be easier to optimize during training. The equation of the addition, as shown in Eq. (7).

(7) $$X_{BasicOUT}=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(X_{main}+X_{shortcut}),$$where $X_{main}$ is the output of the main branch, $X_{shortcut}$ is the output of the shortcut connection, and $X_{BasicOUT}$ is the output of the BasicBlock. $\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}(\cdot )$ represents the ReLU activation function.

Transformer branch

Vision Transformer (ViT) has shown state-of-the-art results in various computer vision applications, such as image classification and segmentation. This motivates us to use ViT to extract global features from EEG feature representation. The structure of the Transformer branch is shown in Fig. 4.

The input of ViT is the spatial-spectral feature representation $S^M=(S_1^M,S_2^M,...,S_B^M)\in {\mathbb{R}}^{H\times W\times B}$. The shape of $S^M$ is $32\times 32\times 5$. $S^M$ is first processed by Stage 0 and segmented into small blocks, each called a patch. Then, each patch is compressed into a one-dimensional vector, which serves as the input for subsequent self-attention calculations. We set $patch\mathrm{\_}size=32$ and $hidden\mathrm{\_}dim=256$. Therefore, $S^M$ with a shape of $32\times 32\times 5$ is divided into five patches of size $(32\times 32)$ each, with a shape of $32\times 32\times 1$. After Stage 0, each patch is compressed into a vector of size $256$, resulting in an output shape of Stage 0 of $5\times 256$.

The features output from Stage 0 are processed through Stage 1, Stage 2, Stage 3, and Stage 4 to generate the T1, T2, T3, and T4 features, respectively. ViT’s Stage 1, Stage 2, Stage 3, and Stage 4 each contain three EncoderBlocks. The EncoderBlock in ViT is designed to establish global context relationships between different positions of the input feature map so that the model can better understand the input feature map. An EncoderBlock comprises of three primary components: multi-head attention, multi-layer perceptron (MLP) block, and layer normalization. Below is a detailed description of each of these components.

Multi-head attention is utilized to perform self-attention operations in different representation subspaces to capture global relationships. Specifically, the Multi-Head Attention first maps the input vector X into three sequences of vectors, namely, $Q\in {\mathbb{R}}^{n\times d_q}$, $K\in {\mathbb{R}}^{n\times d_k}$, and $V\in {\mathbb{R}}^{n\times d_v}$, by three different linear transformations, namely, Query, Key, and Value matrices. Here, $d_q$, $d_k$, and $d_v$ are the dimensions of the Query, Key, and Value vectors, respectively. Then, Q, K, and V are transformed by linear mappings of dimension $d_h$ to obtain multiple heads H, as shown in Eq. (8).

(8) $$hea{d}_i=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q{W}_i^Q,K{W}_i^K,V{W}_i^V),$$where, $W_i^Q\in {\mathbb{R}}^{d_q\times d_h}$, $W_i^K\in {\mathbb{R}}^{d_k\times d_h}$, and $W_i^V\in {\mathbb{R}}^{d_v\times d_h}$ are parameter matrices of the $i$-th head, and $\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(\cdot )$ denotes the dot-product attention mechanism.

Dot-product attention is used to compute the similarity between Q and K and weight V by this similarity, as shown in Eq. (9).

(9) $$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(Q,K,V)=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V,$$where, Q, K, and V represent the query, key, and value vectors, respectively, and $d_k$ is the dimension of the key vector. The function $\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(\cdot )$ is used to normalize the similarity between the query and key vectors, obtaining the attention weights, which are then used to obtain the weighted sum of the value vectors, resulting in the final output.

To enable the model to learn features in different subspaces and increase its expressive power, the multi-head attention concatenates the output vectors of these attention heads together, and then maps them to a new space through a linear transformation layer, as shown in Eq. (10).

(10) $$\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{t}\mathrm{i}\mathrm{H}\mathrm{e}\mathrm{a}\mathrm{d}(Q,K,V)=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(hea{d}_1,\cdots ,hea{d}_h)W^O,$$where $hea{d}_i$ represents the output of the $i$-th attention head, $h$ denotes the number of heads, $\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(\cdot )$ is the operation that concatenates the output of each head, and $W^O$ is the linear transformation matrix for the final output.

The MLP Block refers to a multi-layer perceptron module, also known as a fully connected layer module, usually consisting of two linear layers and a GELU activation function. Specifically, the input to each MLP Block is a tensor processed by the multi-head attention module, with a shape of $N\times L\times D$, where N represents the number of input sequences, L represents the length of the sequence, and D represents the vector dimension at each position. Then, the MLP Block transforms each vector at each position through two fully connected layers and uses GELU as the activation function. The MLP Block can not only improve the expressive power of the model but also ensure that information is not lost during processing. The MLP Block is shown in Eq. (11).

(11) $$\mathrm{M}\mathrm{L}\mathrm{P}(x)=\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}(\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}(\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}(x{W}_1+b_1))W_2+b_2),$$where, $x$ is the input vector, $W_1$ and $W_2$ are the weights of the two fully connected layers, $b_1$ and $b_2$ are the bias terms, $\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}(\cdot )$ represents the GELU activation function, and $\mathrm{D}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{o}\mathrm{u}\mathrm{t}(\cdot )$ represents the Dropout layer, which is used to reduce overfitting.

Layer normalization is used to normalize inputs to improve the stability and training effectiveness in MLP and Multi-Head Attention modules. Specifically, layer normalization normalizes inputs by subtracting the mean and dividing by the standard deviation of the features. This approach effectively combats the problem of vanishing gradients, without reducing network expressivity, thereby enhancing network stability. The formula of layer normalization is shown in Eq. (12).

(12) $$\mathrm{L}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}(\mathrm{x})=\gamma \frac{x-\mu }{\sqrt{{\sigma }^2+\epsilon }}+\beta ,$$where, $x$ represents the input features, $\gamma$, and $\beta$ are learnable scaling and shifting parameters, $\mu$ and $\sigma$ are the mean and standard deviation of the input vector $x$, and $ϵ$ is a very small value used to avoid the denominator being zero.

CIT module

The local features extracted by the CNN branch can provide more fine-grained information, while the global features extracted by the Transformer branch can provide more comprehensive contextual information. To better integrate local and global features and thus improve the accuracy and robustness of the model, we propose a Convolution Interactive Transformer module, which is called the CIT module. The CIT module enables the interaction and fusion of local and global features, thus enhancing the ability of the model to extract both global and local features from EEG spatial-spectral feature representation, leading to improved performance. The overall framework of the CIT module is shown in Fig. 5.

Every CIT module consists of two blocks: the Local to Global (L2G) block and the Global to Local (G2L) block. The CNN feature maps (Stage $k$, where $k=1,2,3$) serve as inputs to the L2G block, which outputs ViT feature maps (Stage $k+1$, where $k=1,2,3$). Similarly, the ViT feature maps (Stage $k$, where $k=1,2,3$) serve as inputs to the G2L block, which outputs CNN feature maps (Stage $k+1$, where $k=1,2,3$). The CIT module can make the local features of the CNN feature map and the global features of the ViT feature map form a coupling state. In other words, global features and local features can interact with each other in the entire feature learning process. The following is a detailed introduction to the L2G block and the G2L block.

In order to improve the ability of the model to extract features from different regions of EEG spatial-spectral feature representation and thereby improve the classification performance, we propose a Convolution Interactive Transformer module, termed the CIT module. The CIT module enables the interaction and fusion of local and global features, thus enhancing the ability of the model to extract both global and local features from EEG spatial-spectral feature representation, leading to improved performance. The overall framework of the CIT module is shown in Fig. 5. Every CIT module consists of two blocks: the Local to Global block, termed L2G, and the Global to Local block, termed G2L. The CNN feature maps (Stage $k$, where $k=1,2,3$) serve as inputs to the L2G block, which outputs ViT feature maps (Stage $k+1$, where $k=1,2,3$). Similarly, the ViT feature maps (Stage $k$, where $k=1,2,3$) serve as inputs to the G2L block, which outputs CNN feature maps (Stage $k+1$, where $k=1,2,3$). The CIT module can make the local features of CNN feature map and the global features of ViT feature map form a coupling state. In other words, global features and local features can interact with each other in the entire feature learning process. The following is a detailed introduction to the L2G block and the G2L block.

Datasets and settings

The study was carried out using the SEED (Zheng & Lu, 2015) and SEED-IV (Zheng et al., 2018) datasets. Both are public EEG datasets used primarily for emotion recognition. The SEED datasets contain EEG data from 15 subjects, who were presented with 15 Chinese film clips. Each clip-viewing process was divided into four stages, including a 5-s start prompt, a 4-min clip period, a 45-s self-assessment, and a 15-s rest period. EEG recordings were conducted three times on each subject, with a 2-week interval between each recording. Each recording session consisted of 15 movie clips, with each clip evoking positive, neutral, or negative emotions. The SEED-IV datasets, an extension of the SEED datasets, include 72 video clips of 2-min duration each, aimed at evoking happy, sad, fear, and neutral emotions in the subjects, who self-evaluated their emotions after watching the video clips. The EEG signals were recorded using the ESI Neuroscan system from 62 channels with a sampling rate of 1,000 Hz, which was downsampled to 200 Hz. The EEG data were filtered using a band-pass filter to remove noise and artifacts, and features such as Power Spectral Density were extracted from each segment in five frequency bands ( $\delta$: $1\sim 4$ Hz, $\theta$: $4\sim 8$ Hz, $\alpha$: $8\sim 14$ Hz, $\beta$: $14\sim 31$ Hz, $\gamma$: $31\sim 50$ Hz). The number of data samples in each class after the windowing operation is shown in Table 1.

We trained and tested the CIT-EmotionNet model using Tesla V100-SXM2-32 GB GPU, and implemented it using the PyTorch framework. The training was conducted using Adam optimizer, and the learning rate was set to 1e−5. The batch size was set to 32, and the dropout rate was set to 0.2. The number of classes to classify for the SEED dataset was 3, while for the SEED-IV dataset, it was 4. The samples were randomly shuffled, and the data were divided into training and testing sets with a ratio of 6:4. The cross-entropy loss function was used. The summary of hyper-parameter settings is as follows.

• Optimizer–Adam.

• Learning rate–1e−5.

• Number of classes–3 (SEED) or 4 (SEED-IV).

• Batch size–32.

• Loss function–cross-entropy.

• Dropout rate–0.2.

Evaluation metrics

The proposed CIT-EmotionNet method will evaluate its performance based on the following metrics: average accuracy (ACC) and standard deviation (STD). The accuracy rate is defined as the ratio of correctly identified positive and negative samples to the total number of samples, as shown in Eq. (23):

(23) $$\mathrm{A}\mathrm{C}\mathrm{C}=\frac{TP+TN}{TP+TN+FP+FN}$$where TP represents the number of predicted positive samples in the positive samples, TN represents the number of predicted negative samples in the negative samples, FP represents the number of predicted positive samples in the negative samples, and FN represents the number of predicted negative samples in the positive samples. The standard deviation is shown in Eq. (24):

(24) $$\mathrm{S}\mathrm{T}\mathrm{D}=\sqrt{\frac{\sum _{i=1}^n{(x_i-\overline{x})}^2}{n-1}}$$

Experimental results and analysis

In order to validate the effectiveness of our proposed model, we compared the proposed model with several baseline methods on the SEED datasets and SEED IV datasets, and briefly introduced each method as follows.

• Support vector machine (SVM) (Zheng & Lu, 2015): A support vector machine that utilizes a linear kernel.

• Dynamic graph CNN (DGCNN) (Song et al., 2018): It represented the spatial relationship between multi-channel EEG signals as a graph, and then extracted features from the graph using dynamic graph convolutional neural networks, resulting in more discriminative EEG signal features.

• Bi-hemispheric discrepancy model (BiHDM) (Li et al., 2020): It used four directed recursive neural networks (RNNs) based on two spatial directions to traverse the electrode signals of two different brain regions, obtaining a deep representation of all EEG electrode signals while maintaining their inherent spatial dependencies.

• Regularized graph neural network (RGNN) (Zhong, Wang & Miao, 2020): It considered the biological topological structure between different brain regions to capture local and global relationships between different EEG channels.

• SST-EmotionNet (Jia et al., 2020): It integrated spatial-spectral-temporal features and used adaptive 3D attention mechanisms to explore discriminative local patterns.

• 3D CNN & PST (Liu et al., 2021): It introduced a 3D CNN-based method that includes a positional-spectral-temporal attention module. This module is composed of positional attention, spectral attention, and temporal attention modules, which respectively explore different brainwave features.

• EEG emotion Transformer (EeT) (Liu et al., 2022): It proposed a transformer-based framework for EEG emotion recognition that exclusively utilizes self-attention blocks.

• Joint-Dimension-Aware Transformer (JDAT) (Wang et al., 2023): It proposed a transformer-based model for EEG emotion recognition. By applying adaptive MSA to multidimensional features, It could focus on different EEG information, including spatial, frequency, and temporal features.

• MD-AGCN (Li, Wang & Lu, 2021): It proposed a method called the Multi-Domain Adaptive Graph Convolutional Network for EEG signal analysis, which utilized the complementary knowledge between different domains of EEG signals and the topological structure of EEG channels.

• 4D-aNN (Xiao et al., 2022): It proposed an attention-based 4D neural network for EEG emotion recognition. It utilized CNN to process the 4D spectral-spatial representation of EEG signals. Meanwhile, it integrated the temporal attention mechanism into the bidirectional long short-term memory to explore the temporal dependency of the 4D spectral-spatial representation.

• MDGCN-SRCNN (Bao et al., 2022): It combined graph convolutional networks (GCN) and convolutional neural networks (CNN). GCN learned spatial features at different levels, while CNN learned abstract features. The fully connected layer was used to fuse shallow spatial features with deep abstract features to identify highly discriminative features for emotion classification.

• CLISA (Shen et al., 2023c): A contrastive learning method is proposed to enhance the generalization ability of the emotion recognition model. This method minimizes subject differences by maximizing the similarity of EEG signal representations from different subjects when exposed to the same emotional stimuli.

• 3D-CNN-ELM (Yuvaraj et al., 2023): It proposed using a three-dimensional convolutional neural network (3D-CNN) to recognize emotions from the spatial-temporal representation of EEG signals. First, the raw EEG signals were transformed into a 3D spatial-temporal representation to better capture spatial and temporal information between electrodes. Then, emotion recognition was conducted using 3D-CNN, leveraging its capability to learn spatial and temporal features.

• EEGformer (Wan et al., 2023): It introduced a transformer-based EEG analysis model called EEGformer. EEGformer performed unified feature extraction and encoding of EEG signals by employing sequence-based transformers for synchronization, localization, and temporal transformation, thereby extracting global features of EEG signals.

Table 2 reports the average accuracy (ACC) and standard deviation (STD) of these methods and the proposed CIT-EmotionNet model for EEG emotion recognition. The SST-EmotionNet utilized a 3D CNN-based method and employed spatial-spectral-temporal features, achieving an ACC of 96.02% and 84.92% on the SEED dataset and SEED IV dataset, respectively. However, the SST-EmotionNet did not consider global features. EeT used a Transformer-based method and thus performed better than CNN-based methods on the large SEED dataset, achieving an ACC of 96.28%. However, EeT ignored local features and performed worse on the smaller SEED IV dataset, achieving an ACC of 83.27%. MDGCN-SRCNN is a combined method based on GCN and CNN that considers the fusion of different features, achieving an ACC of 95.08% and 85.22% on the SEED dataset and SEED IV dataset, respectively. In general, CIT-EmotionNet integrates local and global features well, enabling it to capture valuable features from EEG signals for emotion recognition comprehensively. Compared to the baseline model, CIT-EmotionNet further improves its accuracy. It achieved an ACC of 98.57% and an STD of 1.38% on the SEED dataset and an ACC of 92.09% and an STD of 5.33% on the SEED IV dataset, outperforming the current state-of-the-art methods.

Ablation experiments

In order to validate the effects of different components in our model on the EEG emotion recognition tasks, we performed ablation experiments on both the SEED and SEED IV datasets. Three ablation experiments were conducted. The first ablation experiment was conducted to the effectiveness of the fusion and interaction of local and global features. The second ablation experiment was conducted to evaluate the CIT module, aiming to validate the effectiveness of each component. The third ablation experiment investigates the impact of CIT module quantity on the performance of CIT-EmotionNet.

Ablation experiments on the major components of CIT-EmotionNet

CIT-EmotionNet consists of three main components: the CNN branch, the Transformer branch, and the CIT module. In order to validate the effectiveness of the fusion and interaction of local and global features, we conducted ablation experiments on the major components of CIT-EmotionNet. Table 3 illustrates the impact of different components of CIT-EmotionNet on EEG emotion recognition tasks. “With only CNN branch” indicates the use of only the CNN branch, which means using ResNet alone for the EEG emotion recognition task. “With only Transformer branch” indicates the use of only the Transformer branch, which means using ViT alone for the EEG emotion recognition task. “Baseline” refers to the removal of all CIT modules, and only a simple concatenation fusion is performed on the output of the feature by the CNN branch, and the Transformer branch.

Table 3:

Ablation experiments on the major components of CIT-EmotionNet.

Method	SEED		SEED-IV
	ACC (%) $\uparrow$	STD (%) $\downarrow$	ACC (%) $\uparrow$	STD (%) $\downarrow$
With only CNN branch	74.36	8.72	50.93	18.56
With only Transformer branch	91.77	3.98	82.87	7.18
Baseline	93.39	3.66	87.81	6.63
CIT-EmotionNet	98.57	1.38	92.09	5.33

DOI: 10.7717/peerj-cs.2610/table-3

“With only CNN branch” achieves an ACC of 74.36% and an STD of 8.72% on SEED datasets, while achieving an ACC of 50.93% and an STD of 18.56% on SEED IV datasets. “With only Transformer branch” achieves an ACC of 91.77% and an STD of 3.98% on SEED datasets, while achieving an ACC of 82.87% and an STD of 7.18% on SEED IV datasets. This indicates that Transformer performs better than CNN on the EEG emotion recognition task. “Baseline” achieved an accuracy of 93.39% and a standard deviation of 3.66% on the SEED dataset, while on the SEED IV dataset, the accuracy and standard deviation were 87.81% and 6.63%, respectively. These results indicate that the combination of local and global features contributes to improving the recognition performance of the model.

We present the confusion matrices of the CNN branch, Transformer branch, Baseline, and CIT-EmotionNet models on the SEED dataset in Fig. 6. These matrices illustrate the classification performance of each model across different emotion categories. The experimental results show that CIT-EmotionNet outperforms the CNN branch, Transformer branch, and Baseline models in recognizing negative, neutral, and positive emotions. Furthermore, on the SEED dataset, CIT-EmotionNet demonstrated superior performance in recognizing positive emotions when compared to negative and neutral emotions.

Additionally, in Fig. 7, we present the confusion matrices of the CNN branch, Transformer branch, Baseline, and CIT-EmotionNet models on the SEED IV dataset, illustrating their respective classification performance across different emotion categories. The experimental results demonstrate that CIT-EmotionNet outperforms the CNN branch, Transformer branch, and Baseline models in recognizing sad, neutral, fear, and happy emotions. However, in contrast to recognizing emotions such as sad, neutral, and happy, CIT-EmotionNet exhibits poorer performance in recognizing fear emotions on the SEED IV dataset.

Ablation experiments on the number of CIT modules

To investigate the impact of the number of CIT modules and their corresponding stages on the recognition accuracy of the model, ablation experiments were conducted with varying numbers of CIT modules at different stages on both the SEED and SEED IV datasets. The specific performance results are shown in Fig. 8. “Baseline” means no CIT module is added. “S1” represents adding CIT module only in stage 1. “S2” represents adding CIT module only in stage 2. “S3” represents adding CIT module only in stage 3. “S1 & S2” represents adding CIT module in both stage 1 and stage 2. “S1 & S3” represents adding CIT module in both stage 1 and stage 3. “S2 & S3” represents adding CIT module in both stage 2 and stage 3. It can be seen from the figure that adding CIT module in stage 3 performs better on SEED dataset, while adding CIT module in stage 2 performs better on SEED IV dataset. Meanwhile, adding CIT module in all stages, which is the strategy of Ours, performs best on both SEED datasets and SEED IV datasets.