A comprehensive review of deep learning in EEG-based emotion recognition: classifications, trends, and practical implications

Summary of previous reviews

Roy et al. (2019) underscores the potency of deep learning in acquiring enhanced feature representations directly from raw EEG data. The authors meticulously reviewed 154 deep learning-based EEG articles published from January 2010 to July 2018 to distill the prevailing trends and methodologies in this context. Their objective was to glean insights that could inform future research endeavors. Notable areas of exploration spanned epilepsy, sleep studies, brain-computer interfaces, cognitive investigations, and emotion detection. Remarkably, the analysis revealed a dominant presence of convolutional neural networks (CNNs) in 40% of the studies, recurrent neural networks (RNNs) featuring in 13% of the cases, while a significant portion of these studies—almost half—focused on training models utilizing either raw or preprocessed EEG time series data.

Suhaimi, Mountstephens & Teo (2020) delves into the progress of EEG-based emotion recognition between 2016 and 2019. It highlights various factors, including the type and delivery of emotional stimuli, the sample size of studies, EEG hardware employed, and machine learning (ML) techniques utilized. The article introduces a novel approach involving virtual reality (VR) for presenting stimuli, drawing motivation from a review of VR research within the emotion recognition domain.

Rahman et al. (2021) conducts a comprehensive review of published research utilizing EEG signal data to explore potential links between emotions and brain activity. The article elucidates the theoretical underpinnings of basic emotions and outlines prominent techniques for feature extraction, selection, and classification. A comparative analysis of study outcomes is provided, alongside a discussion of future directions and significant challenges anticipated in the development.

Li et al. (2022d) offers a comprehensive analysis from a researcher’s perspective. This review delves into the intricacies inherent to the science of the field, examining the psychological and physiological foundations, elucidating distinct conceptual pathways and theoretical frameworks, elucidating the driving forces behind such investigations, and rationalizing the research and application of these methodologies.

Dadebayev, Goh & Tan (2022) offers a critical assessment of consumer-grade EEG devices and contrasts them with their research-grade counterparts. They highlight critical aspects of EEG-based emotion recognition research and identify primary challenges in system development, focusing on data collection and machine learning algorithm performance with commercial EEG devices.

Jafari et al. (2023) focus lies in utilizing deep learning (DL) methodologies to recognize emotions from EEG signals. The article extensively engages with the pertinent academic literature, engaging in an in-depth examination of the complexities inherent in EEG-based emotion recognition. It emphasizes the promise held by DL techniques to address and ameliorate these challenges. Additionally, the article outlines potential directions for future studies in utilizing DL techniques for recognizing emotions. A particularly auspicious trajectory for future investigations involves the exploration of system-on-chip (SoC) architectures leveraging field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs) for enhancing emotion recognition from EEG signals. Finally, in its concluding segment, the article summarizes its primary findings, affirming the emergence of DL techniques as promising solutions for EEG-based emotion recognition. It also presents a comprehensive review of studies employing DL methodologies.

Khare et al. (2023) extensively examines emotion recognition techniques, emphasizing the utilization of diverse information sources such as questionnaires, physical cues, and physiological signals, including EEG, ECG, galvanic skin response, and eye tracking. The review encompasses various physical cues like speech and facial expressions, alongside a comprehensive survey of emotion models and stimuli used to evoke emotions. The article systematically analyzes 142 journal articles following PRISMA guidelines, offering insights into existing research, available datasets, and potential challenges. Challenges include variable signal lengths in emotion recognition studies due to diversity in system specifications, hindering trust in model decisions. Existing automated systems face trust issues due to discrepancies with prior knowledge and limited real-time support, necessitating transparent model explanations. Future research directions suggest leveraging federated meta-learning to create accurate and versatile models across diverse datasets for specific applications.

In recent years, the continuous evolution of deep learning has led to the emergence of an increasing number of models designed for EEG-based classification. Consequently, classification accuracy has steadily improved over time. While some researchers have generalized and summarized these methods, they have typically approached the topic from diverse technical perspectives (Prabowo et al., 2023; Vempati & Sharma, 2023). These approaches provide valuable insights into the theoretical underpinnings and research motivations. Table 1 summarizes the reviews mentioned above for reference succinctly. However, there remains a gap in the literature regarding the comprehensive categorization and classification of this field’s diverse directions and methodologies. This gap poses a challenge for newcomers to the field who seek a quick understanding of its developmental direction. Hile’s previous researchers have attempted to classify methods within emotion recognition based on EEG signals. However, these attempts have often needed more clarity in perspective and have not sufficiently explained the rationale behind their classification.

To tackle the issues above, this article introduces a novel classification approach aimed at effectively categorizing existing studies in the domain of EEG-based emotion recognition. Given the intrinsic variability in EEG signals across individuals, this categorization method delineates two specific directions: subject-independent and subject-dependent. These directions correspond to research involving classification across individuals and research that does not cross individual boundaries.

We can effectively synthesize the underlying research principles and motivations by categorizing ongoing research into these two domains. Within the subject-independent direction, researchers aim to enhance model generalization and robustness. Their primary objective is to maximize the extraction of standard essential information, mainly when dealing with distinct source and target domains. This direction significantly emphasizes the model’s ability to generalize effectively and maintain robustness across differing domains. Conversely, in the subject-dependent direction, researchers are dedicated to improving the model’s feature extraction capabilities to enhance classification accuracy. The aim is to extract and incorporate a comprehensive array of sample features, thus elevating classification accuracy. Moreover, this article comprehensively overviews prevailing methods and models applied in diverse domains. Finally, it offers a cross-disciplinary perspective on the field’s practical relevance and potential future applications.

Contribution of the review

The primary contributions of this article are as follows:

• Novel classification framework: This article introduces a well-structured classification method based on research directions, encompassing subject-independent and subject-dependent approaches. It provides a clear rationale for classification and elucidates the distinct research objectives within each direction.

• Comprehensive method summaries: Given the inherent variability in research objectives across diverse directions, this article presents a thorough synthesis of scholarly works spanning the field’s six years from 2018 to 2023. Emphasizing distinct modeling approaches, it furnishes a comprehensive overview of methodologies pertinent to each research direction.

• Detailed dataset comparison: This article explores both mainstream and state-of-the-art datasets, offering a comparative analysis of these four prominent datasets.

• Comprehensive overview of EEG signal-based classification: This article provides a comprehensive summary of the recent research developments in EEG-based classification. It further delves into the practical significance and potential applications of this field. In addition, the article suggests several methods that can be integrated with other domains to enhance the value and utility of EEG-based classification.

Target audience of the review

This review is dedicated to introducing innovative categorization methods for emotion classification using EEG signals. It is intended for a diverse audience, including professionals from various domains such as human–computer interaction, brain-computer interfaces, emotion recognition, affective computing, and EEG studies. We recommend this review to the following readers:

• Researchers and scholars are interested in gaining insights into publicly available datasets for emotion recognition through EEG signals. This review provides an overview of these datasets, highlighting their commonalities and distinctions.

• Researchers and scholars are seeking to comprehend the diverse research directions in emotion recognition via EEG signals. It delves into the core concepts of different research avenues, allowing readers to grasp their distinct approaches and objectives.

• Researchers and scholars are eager to explore the future trajectory of emotion recognition using EEG signals and its potential integration with other related fields. This review offers valuable perspectives on the evolving domain’s prospective developments and interdisciplinary connections.

Organization of the review

The article is organized as follows: In ‘Survey methodology’, we expound on our meticulous approach to the impartial and comprehensive selection and evaluation of pertinent literature. We provide insights into the classification, assessment, and interrelation of analytical studies, all undertaken by the taxonomy proposed within this article. ‘Task and datasets’ is dedicated to elucidating the principal tasks within EEG-based sentiment analysis. It also extensively analyzes mainstream and cutting-edge datasets, delving into their shared characteristics and distinctions. ‘EEG Signal processing’ outlines the preliminary steps preceding the design of EEG signal-based models for sentiment analysis, emphasizing preprocessing and feature extraction procedures applied to raw EEG signals. ‘Machine learning approaches’ delves into the nascent stages of the field, wherein traditional machine learning models were initially employed for classification development. Within ‘Classification and methods’, we outline the classifications introduced in this study. Furthermore, we provide succinct summaries and insightful analyses for each classification and an overview of the pertinent literature. ‘Significance and applications’ explores the field’s relevance, shedding light on prospective integration opportunities with other domains. We proffer suggestions for the fusion of this field with complementary research areas. In ‘Discussion’, we discussed some of the field’s current challenges and future directions. ‘Conclusion’ offers a comprehensive summary of the entire text, meticulously scrutinizing the identified challenges and charting a path forward for future investigations.

Classification tasks

Emotion recognition based on EEG signals encompasses many tasks, including data acquisition, preprocessing, feature extraction, and classification model design. This article predominantly focuses on classification model design, the prevailing pursuit in current research. Prior studies have made significant advancements in preprocessing raw data and extracting relevant features. Consequently, recent research efforts have shifted their focus toward the design of classification models.

The domain of classification model design involves leveraging the principles and models of deep learning to learn from preprocessed EEG signals. This enables the model to analyze signals, effectively enhancing emotion recognition accuracy. Furthermore, it empowers the model to predict emotions in unlabeled EEG signals, facilitating its application in downstream tasks.

Datasets

In the realm of EEG emotion recognition, three publicly accessible mainstream datasets currently stand out: DEAP (Koelstra et al., 2011), SEED (Duan, Zhu & Lu, 2013; Zheng & Lu, 2015), and SEED-IV (Zheng et al., 2018). These datasets are widely recognized and frequently employed by researchers in EEG emotion recognition for classification tasks. As the field of human–computer interaction continues to advance, more researchers are engaging in emotion recognition based on EEG signals, leading to new datasets like DENS (Asif et al., 2023). The subsequent section provides a brief overview of these commonly used datasets.

The DEAP dataset comprises data collected from 32 subjects, evenly split between 16 males and 16 females. Each subject participated in a single experiment, viewing 40 music videos. The experimental procedure for each subject included a 3-second baseline recording, followed by viewing a 60-second music video. Afterward, the subjects spent 15 seconds completing the Self-assessment Manikins (SAM) scale, which includes four dimensions: valence, arousal, dominance, and liking. Subsequently, the subjects watched another 60-second music video.

The SEED and SEED-IV datasets were provided by the BCMI laboratory at Shanghai Jiao Tong University and included data from 15 subjects, comprising seven males and eight females. The subjects have an average age of 23.27 years, with a standard deviation of 2.37 years. The experimental design for these datasets involves each subject participating in three separate trials spaced one week apart. In each trial for every subject, there is an initial 5-second hint to signal the commencement of the trial, followed by a 4-minute movie clip, a 45-second self-assessment period, and concluding with a 15-second rest period.

The DENS dataset encompasses data from 40 participants with an average age of 23.3 years and a standard deviation of 1.25 years. Each participant was subjected to a single experiment, which involved viewing 11 music videos. These videos included nine emotional stimuli and two non-emotional stimuli. The experimental protocol for each participant followed a consistent sequence: an initial 80-second baseline recording, followed by viewing a 60-second video segment. Subsequently, participants completed the Self-assessment Manikins (SAM) scale, which measured various emotional dimensions, including valence, arousal, dominance, liking, familiarity, and relevance. Finally, participants proceeded to watch another 60-second video segment. Table 2 compares the commonalities and differences between these four datasets.

Preprocessing

During EEG data collection, it is imperative to acknowledge the presence of various forms of noise and artifacts stemming from intrinsic subject-related factors, experimental environmental conditions, or equipment-related influences. Consequently, raw data collected from EEG recordings cannot be directly utilized for analysis and classification purposes. Noteworthy among these disturbances are electrooculographic, electrocardiographic, electromyographic, and industrial frequency artifacts. Electrooculographic artifacts typically manifest below 4 Hz, attributed to eye movements or blinking. Electrocardiographic artifacts, occurring at approximately 1 Hz, result from cardiac contractions. Electromyographic artifacts, characterized by frequencies exceeding 25 Hz, stem from muscular activities, particularly in temporal, prefrontal, and cervical regions. Industrial frequency interference, typically at 50 or 60 Hz, originates from utility sources. Hence, mitigating these noise and artifacts is imperative for accurate analysis and classification of EEG signals, facilitating extracting meaningful insights from the recorded data. Consequently, removing such disturbances is essential to ensure the integrity and reliability of EEG signal analysis and classification endeavors.

Given the criticality of mitigating noise and artifacts in EEG signals, numerous studies focus on preprocessing techniques. Commonly employed methods for artifact removal encompass regression methods, wavelet transform, filtering, principal component analysis (PCA), and independent component analysis (ICA). The regression method (Klados et al., 2011), a traditional approach in EEG preprocessing, is versatile, operating effectively in both the time and frequency domains (Whitton, Lue & Moldofsky, 1978). Wavelet transformation (Kumar et al., 2008) facilitates the conversion of time-domain information into both temporal and frequency-domain representations, enabling artifact removal through subsequent thresholding techniques. Various filtering methodologies, such as adaptive filtering (Correa et al., 2007), Wiener filtering (Somers, Francart & Bertrand, 2018), and high-pass filtering (Winkler et al., 2015), offer additional avenues for noise reduction during preprocessing. PCA (Berg & Scherg, 1991) relies on eigenvalues derived from the covariance matrix, providing a powerful means to extract meaningful features while attenuating artifacts. ICA (Jung et al., 1998), an extension of PCA, further enhances artifact removal capabilities by decomposing EEG signals into statistically independent components. Collectively, these advanced techniques signify significant advancements in EEG preprocessing, furnishing researchers with many robust tools to remove noise and artifacts from raw EEG signals effectively.

Feature extraction

Initially, EEG research primarily focused on the time domain of EEG signals, resulting in the extraction of significant features but yielding suboptimal classification outcomes. However, with the evolution of research methodologies, there has been a shift towards acknowledging and incorporating spatial and frequency domain information and exploring joint information integration.

In the time domain, methods such as the Higher-Order Crossing (HOC) method (Petrantonakis & Hadjileontiadis, 2009) and Hjorth features (Patil, Deshmukh & Panat, 2016) have garnered widespread adoption due to their efficacy. Regardless of the number of channels utilized for EEG signal acquisition, be it 32, 62, or 128, it is evident that there exists a spatial relationship among these channels, reflective of the underlying neural architecture. Consequently, this spatial relationship must be noticed during feature extraction. In spatial domain processing, common spatial patterns (CSP) (Wang, 2011) and hierarchical discriminant component analysis (HDCA) (Alpert et al., 2013) techniques are commonly employed to capture spatial dependencies effectively. The frequency domain information, recognized as pivotal in characterizing EEG signals, has garnered increasing attention from researchers. Utilizing techniques like the Fast Fourier Transform (FFT) (Murugappan & Murugappan, 2013) to segment EEG signals into distinct frequency bands (δ, θ, α, β, γ), researchers then analyze these segments using power spectral density (PSD) (Zheng & Lu, 2015; Wang, Nie & Lu, 2014) or differential entropy (DE) (Duan, Zhu & Lu, 2013) to extract frequency domain features. By incorporating spatial and frequency domain information alongside traditional time-domain analyses, researchers can gain a more comprehensive understanding of EEG signals, thereby enhancing the efficacy of feature extraction and classification methodologies.

Subject-dependent

The predominance of research on the subject-dependent approach in EEG-based emotion recognition can be attributed to the profound individual variability intrinsic to EEG signals. This research direction has garnered extensive attention, primarily due to the challenges associated with subject-independent classification, its significant practical importance, and real-world applications. As a result, many researchers have devoted their efforts to the subject-dependent paradigm. Research within this direction can be delineated into two primary phases: the initial phase primarily focused on classification using exclusively time-domain features. Subsequently, the second phase emerged, embracing a more comprehensive feature extraction methodology encompassing the time, spatial, and frequency domains. The technical diagram for this direction is visually represented in Fig. 4.

EEG signals, notable for their high temporal resolution characteristics, were initially introduced into the domain of natural language processing (NLP) through their integration into deep learning models. Initially, sequence data processing in deep learning predominantly relied on recurrent neural network (RNN); however, RNN often grapples with challenges such as vanishing or exploding gradients, particularly when confronted with lengthy sequences. Consequently, long short-term memory (LSTM) networks emerged as a solution for effectively processing prolonged sequence data. Applying deep learning techniques to EEG information commenced with classification tasks employing RNN and LSTM networks.

As the EEG field progressed, many scholars explored the spatial and frequency domain characteristics inherent in EEG signals. In parallel with the evolution of deep learning methodologies, CNN was introduced and gradually integrated into the domain of NLP. Similarly, CNN found application in EEG signal processing for tasks such as emotion recognition. CNN architectures excel in capturing local and hierarchical features within data, rendering them highly adept at processing EEG signals. The emergence of the Transformer architecture marked another significant advancement. Characterized by its self-attention mechanism capable of capturing long-range dependencies within sequences, the Transformer architecture shifted the paradigm. The attention mechanism embedded within Transformer networks facilitates an enhanced focus on EEG features. Consequently, many networks leveraging Transformer architectures for EEG emotion recognition have surfaced in recent years.

Subject-independent

With the ongoing advancements in human–computer interaction and brain-computer interfaces, a growing body of research focuses on EEG signals. Increasingly, researchers are turning their attention to the development of subject-independent emotion recognition algorithms. This research direction poses more significant challenges compared to its predecessor. The challenges stem from inherent individual differences, including variations in physiological aspects such as brain structure, head size, and unique emotional responses to the same stimuli. Consequently, subject-independent research becomes particularly intricate due to these inter-individual differences.

Nonetheless, the demand for large-scale datasets and contemporary applications necessitates developing models independent of individual idiosyncrasies. This need is increasingly observed in diverse fields, including large-scale studies in sociology and psychology, initial diagnostics of mental health disorders, and the implementation of intelligent recommender systems. These applications rely heavily on the model’s generalization capability, enabling it to operate independently of individual variations.

In the design of neural networks and models for subject-independent research, a key distinction is observed in dataset partitioning. Unlike the subject-dependent approach, where datasets are divided collectively, in the subject-independent paradigm, datasets are segregated for individual participants. In this approach, the training and testing sets are distinctly associated with different individuals, framing the problem as a solution to the Source Data and Target Data Mismatch issue. This mismatch arises due to the substantial individual variability inherent in EEG signals. In essence, the subject-independent direction strives to address the inconsistency between the training and testing data, seeking to extract a limited set of standard features rather than individualized characteristics. Therefore, this research direction presents a formidable challenge compared to the preceding one.

In recent years, there has been a growing trend among researchers to integrate domain adversarial and domain adaptation models into subject-independent emotion recognition algorithms. In various fields, these methodologies are frequently regarded as interchangeable due to their shared characteristics and mutual objective of facilitating the model’s acquisition of domain-agnostic feature representations. Domain adaptation primarily centers on mitigating distributional variance, whereas domain adversarial methods employ adversarial training techniques. Within the domain of EEG emotion recognition, these approaches may be delineated as distinct yet interconnected strategies. Figure 5 succinctly delineates the fundamental principles underlying these two avenues of research.