Systematic review: progress in EEG-based speech imagery brain-computer interface decoding and encoding research

Survey methodology

This review utilized a method called PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), which is a protocol for conducting systematic reviews and analyses (Page et al., 2021). By using PRISMA to identify and locate relevant studies, the data collection process for this review was significantly streamlined.

The search was completed on March 8, 2025. We conducted queries using the following keywords on Web of Science, Scopus, IEEE Xplore, PubMed, ScienceDirect, and Google Scholar to gather relevant literature: (“brain-computer interface” AND (“speech imagery” OR “covert imagery” OR “inner speech” OR “imagined speech”) AND (“deep learning” OR “machine learning” OR “DL” OR “ML”) AND (“electroencephalogram” OR “EEG”)). These keywords were sufficient to retrieve all pertinent studies.

Studies were excluded based on the following criteria:

After retrieving the search results, we excluded duplicate articles across databases and then applied the following criteria to filter out unsuitable studies from the remaining articles. Subsequently, we thoroughly read the full texts of the remaining studies.

(1) EEG-only studies: Studies using multimodal datasets were excluded.

(2) Language: Articles not written in English were excluded.

(3) Task type: Neural encoding and decoding tasks not exclusively based on EEG speech imagery were excluded.

(4) Time frame: Studies published before 2015 were excluded.

Ultimately, 125 highly relevant articles were selected for this literature review analysis. The distribution of these articles across brain connectivity, paradigms, neural encoding, and neural decoding algorithms (machine learning and deep learning) is illustrated in Fig. 3.

Vowel/vowel consonant combination

In speech imagery-based BCI research, vowels are frequently used as core materials due to their simplicity in articulation, stability in acoustic characteristics, and ease of perception and measurement. Researchers have extensively explored the potential of EEG signals in decoding vowel information through diverse methods, progressively advancing BCI technology. DaSalla et al.’s (2009) pioneering work laid a solid foundation for vowel imagery recognition. By recording EEG signals from healthy subjects imagining the English vowels /a/ and /u/, they applied common spatial pattern (CSP) to design spatial filters and used nonlinear SVM for classification, achieving an accuracy rate of 68% to 78%. This study effectively addressed the challenge of extracting features related to vowel imagery from EEG signals, providing crucial methodological guidance for subsequent research. Building on this, Min et al. (2016) employed the Extreme Learning Machine (ELM) algorithm combined with sparse regression feature selection techniques to classify single-trial EEG signals from five subjects imagining vowels. They found that ELM and its variants (ELM-R, ELM-L) outperformed SVM and linear discriminant analysis (LDA) in the gamma frequency band (30–70 Hz), achieving a maximum accuracy of 99%. This work offered new insights into optimizing classification algorithms for silent speech BCIs. Chengaiyan & Anandan (2022) further refined vowel classification in speech imagery using EEG signal decomposition techniques and advanced machine learning methods such as multi-class SVM (MSVM) and RF. Their study demonstrated the reliability of brain connectivity estimators and machine learning techniques in vowel recognition, providing new empirical support for the development of speech imagery recognition technology. This series of studies not only progressively improved vowel imagery classification accuracy but also deepened our understanding of EEG signal processing and vowel imagery recognition mechanisms.

As research on vowel imagery has deepened, researchers have increasingly recognized the need to incorporate consonant processing into BCI systems to more comprehensively simulate natural speech communication. Given that actual speech communication involves not only vowels but also consonants, particularly the prevalence of CVC structured words in the language, researchers have begun exploring how to decode combinations of consonants and vowels in BCI systems. This direction represents a significant step forward in the development of speech imagery-based BCI systems.

The work by Sandhya et al. (2018) marks an early exploration in this field. They investigated relative power changes during multiple speech imagery tasks, with a particular focus on frequency bands in EEG signals. By analyzing EEG signals from healthy subjects imagining and articulating CVC words, they found that the theta band exhibited higher relative power during imagined speech, while the alpha band dominated during actual speech. This suggests that different speech imagery tasks may activate distinct frequency bands in the brain. Building on the work of Sandhya et al. (2018), Chengaiyan et al. (2018) further explored the power and phase synchronization effects in EEG signals. Using the PLV to measure phase synchrony between brain regions during speech imagery, they discovered that the frontal and temporal electrodes in the left hemisphere showed the highest phase locking in the alpha and theta bands. This work highlights the critical role of functional connectivity between specific brain regions in decoding tasks during speech imagery. Sandhya et al. (2015) focused on analyzing the neural correlations of brain regions during the speech imagery of CVC words. Using brain connectivity estimators such as EEG coherence, partial directed coherence (PDC), and directed transfer function (DTF), they found that the frontal and temporal regions of the left hemisphere were more active during both imagined and actual articulation. This study provides new insights into the neural interactions in the brain during speech imagery and offers neurological support for decoding CVC words. Chengaiyan, Retnapandian & Anandan (2020) further advanced this direction by integrating deep learning techniques, such as recurrent neural networks (RNN) and deep belief networks (DBN), with brain connectivity estimators to focus on vowel recognition from EEG signals. In their experiments, DBN demonstrated higher classification accuracy compared to RNN, providing strong empirical support for the use of deep learning techniques in speech imagery decoding.

From the initial CSP and SVM-based classification methods to advanced techniques involving deep learning networks (such as CNN, RNN, DBN, and capsule networks (CapsNet)) and brain connectivity analysis, researchers have continuously deepened their understanding of vowel and vowel-consonant combination information in EEG signals. The development of these technologies has not only complemented each other methodologically but has also made significant progress in improving classification accuracy and the practicality of BCI systems.

Chinese characters/words/long words

In studies using Chinese characters as materials for pronunciation imagery, directional characters such as “左(left)”, “右(right)” have become the primary focus. As research has progressed, the emphasis has shifted from simple characters to those with tonal and rhythmic features, and from single speech imagery tasks to those combining auditory and visual conditions. In research on words and word pairs, although the materials differ, the mental tasks are similar. Notably, the use of local ear EEG as an alternative to global EEG has emerged as a novel approach.

Wang et al. (2016) made pioneering contributions in this field by introducing and applying the novel concept of “speech imagery” to BCI systems. They conducted a study to explore the impact of silent reading on mental tasks within a BCI system. They integrated speech imagery (i.e., silently reading Chinese characters) into mental tasks and found that adding speech imagery significantly improved task accuracy, with the average accuracy increasing from 76.3% to 82.3%. They also evaluated the time stability of EEG signals using Cronbach’s alpha, revealing that the tasks incorporating speech imagery exhibited higher signal stability, providing a more reliable signal source for BCI systems. Furthermore, the classification of Mandarin tones and word pairs has also received widespread attention. Li & Chen (2020) conducted in-depth studies on Mandarin tone classification, analyzing factors that affect the classification accuracy of EEG signals. This work laid the foundation for the application of BCI technology in Mandarin speech synthesis systems and contributed to the theoretical development of speech-imagery-based BCI systems. Borirakarawin & Punsawad (2023) explored hybrid BCI systems that combine auditory stimuli with speech imagery, while Zhang, Li & Chen (2020) further explored tone classification under audiovisual conditions, and their experimental results showed that the introduction of audiovisual conditions significantly improved classification accuracy, demonstrating that multimodal inputs can effectively enhance BCI system performance and provide new experimental evidence for Mandarin tone-based BCI applications.

In research on words and word pairs, innovative algorithms and techniques have emerged continuously. Mahapatra & Bhuyan (2022) achieved multi-class imagined speech (vowels and words) classification based on EEG signals by constructing a deep model that integrates temporal convolutional networks (TCN) with CNN, combined with discrete wavelet transform (DWT) preprocessing. They achieved an overall accuracy of 96.49% on data from 15 participants, providing an efficient decoding solution for non-invasive silent brain-computer interfaces. Pan et al. (2023) employed a light gradient boosting machine (LightGBM) for feature classification in Chinese character speech imagery BCI systems, and the results showed that LightGBM was more accurate and applicable than traditional classifiers. Tsukahara et al. (2019) explored EEG frequency components related to speech imagery and found significant ERS in the left hemisphere’s alpha band under speech imagery conditions, emphasizing the importance of selecting appropriate electrode positions for EEG frequency component recognition. Kaongoen, Choi & Jo (2021) were the first to explore BCI systems based on ear EEG (ear-EEG), providing initial empirical support for the application of ear-EEG in the BCI field.

These studies not only explored various EEG signal processing and feature extraction methods but also experimented with multiple classifiers to improve the decoding accuracy of EEG signals. Furthermore, they considered the complexity of speech imagery, such as semantics, rhythm, and brain activity under different stimulus conditions. These efforts are particularly significant in optimizing BCI system design, improving signal decoding accuracy, and achieving effective classification of Mandarin tones and word pairs.

Signal preprocessing

EEG signal preprocessing involves filtering out irrelevant or interfering signals from the raw EEG data acquired by the device and selecting the signal channels most relevant to speech imagery from specific brain regions. Due to the low signal-to-noise ratio (SNR) of EEG signals, artifacts often occur during acquisition. Therefore, current research focuses on using techniques such as filtering, wavelet transforms, adaptive filters, and statistical methods to remove noise and artifacts, thereby improving signal quality. Jahangiri & Sepulveda (2017) found that the alpha band (8–13 Hz) and beta band (14–30 Hz) yield higher classification performance in speech imagery classification tasks. Consequently, during time-domain and time-frequency analysis, signals are typically filtered within the 8–30 Hz range, with different bandpass filters selected based on specific requirements. Moattari, Parnianpour & Moradi (2017) proposed using independent component analysis (ICA) based on higher-order non-Gaussianity for the source separation stage of preprocessing. Nitta et al. (2023) extracted linguistic representations of Japanese vowels using principal component analysis (PCA) and structural modeling (SM), and demonstrated the feasibility of extracting linguistic features from EEG signals by classifying them with a CNN. Sree, Kavitha & Divya (2023) evaluated various preprocessing methods for EEG signals based on speech imagery. Through a comprehensive comparison of different preprocessing techniques, they identified the optimal approach based on mean squared error (MSE) and peak signal-to-noise ratio (PSNR).

Feature extraction

The feature extraction stage is the core of SI-BCI neural encoding. This process involves extracting useful information from the acquired EEG signals, which is then used in subsequent neural decoding to analyze brain states. Feature extraction algorithms can be categorized into spatial domain methods, time domain methods, frequency domain methods, time-frequency domain methods, and nonlinear feature extraction methods.

In the spatial domain, the main research trend has shifted from traditional CSP and their variants to methods such as Mel-frequency cepstral coefficients (MFCCs). Overall, new methods in the spatial domain are continuously emerging. Wang et al. (2021) proposed a feature extraction method combining causal networks and CSP, achieving higher classification accuracy using a binary quantum particle swarm optimization-based extreme learning machine. Huang et al. (2022) introduced a novel algorithm called sub-time window filter bank common spatial pattern (STWFBCSP), aimed at improving sequence encoding-based active BCIs. By subdividing time windows and applying multi-frequency filtering, STWFBCSP can extract more refined features, thereby enhancing classification accuracy. The final average classification accuracy of STWFBCSP reached 84.87%, compared to 70.49% for the traditional CSP algorithm, demonstrating its superior performance and highlighting the significant potential of time window subdivision and multi-frequency analysis in improving BCI performance. Alizadeh & Omranpour (2023) proposed a novel multi-class CSP feature extraction method, feeding the extracted features into an ensemble learning classifier (including logistic regression, K-nearest neighbors, decision trees, SVM, and Gaussian naive Bayes). After multiple rounds of cross-validation, the effectiveness of the proposed algorithm was confirmed. This combined validation approach effectively inspires algorithmic fusion and innovation. Table 1 illustrates CSP combined with other feature classification algorithms.

Cooney, Folli & Coyle (2018) evaluated the impact of three feature sets—linear, nonlinear, and MFCCs—on classification performance. The study found that MFCC features performed best in both decision tree and SVM classifiers, with SVM achieving an average accuracy of 20.80% on MFCC features, significantly outperforming linear (15.91%) and nonlinear (14.67%) features. This validates the effectiveness of MFCCs in capturing acoustic differences in speech. Martin et al. (2016) introduced the dynamic time warping (DTW) algorithm to address temporal irregularities in speech production, improving the classification performance of SVM. Notably, this marked the first successful classification of individual words during speech imagery tasks. Wu & Chen (2020) proposed a method for extracting temporal envelope features from EEG signals during feature extraction and directly classified these features using SVM. Although the final performance was somewhat lacking compared to other methods, this work explored the relationship between temporal envelope features and speech. Table 2 illustrates other feature extraction algorithms in the spatial domain.

Time-domain methods generally involve selecting features such as the mean, variance, kurtosis, and skewness of signals from each channel. Research in this area primarily focuses on extracting features using wavelet transforms and wavelet decomposition for signals of different frequencies. Idrees & Farooq (2016a, 2016b) proposed a dual-pronged strategy: classifying using simple features in the time domain while extracting EEG signal features from the beta, delta, and theta rhythms using wavelet decomposition. This approach demonstrated the effectiveness of time-domain and frequency-domain features in vowel imagery classification, further expanding the potential of EEG signal processing techniques in speech imagery tasks. Saji et al. (2020) advanced this research by classifying vowel signals using wavelet transforms and multiple classifiers, extracting features such as mean, standard deviation, and band power. This study not only confirmed the effectiveness of EEG signal processing techniques in improving vowel imagery classification accuracy but also provided new empirical support for speech imagery-based BCI technology.

Frequency-domain methods have been more deeply explored in areas such as PSD, DWT, and MFCCs. Cooney, Folli & Coyle (2018) analyzed the EEG signals of imagined speech from 14 participants in the Kara One dataset, evaluating the impact of three feature sets—linear, nonlinear, and MFCCs—on classification performance. The study found that MFCC features performed best in both decision tree and SVM classifiers, with SVM achieving an average accuracy of 20.80% on MFCC features, significantly outperforming linear (15.91%) and nonlinear (14.67%) features. This research demonstrated that MFCC-based feature extraction can significantly improve the accuracy of imagined speech decoding, providing important methodological support for the development of SI-BCIs. Time-frequency domain methods combine information from both the time and frequency domains, commonly using techniques such as STFT and WT. Pan et al. (2022) investigated the impact of imagined syllable rhythms on EEG amplitude spectra, offering new insights for the development of speech imagery-based BCIs. Kamble, Ghare & Kumar (2022) preprocessed EEG signals through bandpass filtering and downsampling in the time domain, combined with time-frequency conversion using the smoothed pseudo-Wigner-Ville distribution (SPWVD). This mapped 1D signals into 2D time-frequency images, enabling optimized CNNs to automatically extract multi-scale time-frequency features. This approach achieved efficient speech imagery classification with a maximum accuracy of 94.82%, validating the advantages of SPWVD in time-frequency resolution and cross-term suppression. It provided a time-frequency analysis-based method for SI-BCI systems.

Nonlinear feature extraction methods refer to the use of deep learning to automatically extract features for subsequent classification tasks. Unlike traditional machine learning, deep learning-based feature extraction methods are often not separable from feature classification methods. This is primarily because, in contrast to deep learning, traditional machine learning requires manual intervention during feature extraction to transform raw data into meaningful features. The core focus of deep learning research lies in developing more efficient and convenient algorithms for feature extraction while maximizing final classification performance. Wang et al. (2022) proposed a parallel CNN model based on a multi-band brain network, utilizing correlation coefficients and phase-locking values to describe inter-channel synchrony and correlation. This approach effectively captures EEG signal synchronization and correlation features, leading to improved classification accuracy. Ramirez-Quintana et al. (2023) introduced a novel deep capsule neural network, CapsVI, designed to recognize vowel imagery patterns from EEG signals. Their model achieved an average accuracy of 93.32%, setting a new benchmark for English vowel recognition. Retnapandian & Anandan (2023) focused on identifying phonemes corresponding to vowels from EEG signals. They trained a RNN using multi-trial data and compared the classification performance between single-trial and multi-trial datasets. By incorporating refined feature extraction techniques such as wavelet decomposition and multifractal analysis, their study further enhanced the classification capabilities of the model, laying the foundation for the development of efficient and precise speech imagery-based BCI systems. Macías-Macías et al. (2020) developed sCNN and sCapsNet models based on spectro-temporal receptive fields (STRFs). Notably, sCapsNet demonstrated superior performance in vowel classification, highlighting the significant potential of deep learning in vowel imagery classification.

In addition to the commonly used feature extraction algorithms mentioned above, several novel methods have also been applied to SI-BCI feature extraction. Sikdar et al. (2018), Sikdar, Roy & Mahadevappa (2018) explored the roles of multifractal and chaotic parameters in different imagery tasks for International Phonetic Alphabet (IPA) vowel recognition, providing new analytical methods for identifying brain regions activated during different vowel imagery processes. Martin et al. (2016) utilized high-gamma band (70–150 Hz) temporal features and a SVM model, introducing nonlinear temporal alignment via an SVM kernel, achieving 88% classification accuracy in a binary classification framework. Nguyen, Karavas & Artemiadis (2017) proposed a novel approach based on covariance matrix descriptors, which reside on the Riemannian manifold, and employed a relevance vector machine (RVM) for classification. This method was tested on EEG signals from multiple subjects and performed well across various categories, including imagined vowels, short words, and long words, with classification accuracy significantly exceeding chance levels in all cases. Kalaganis et al. (2023) leveraged approximate joint diagonalization (AJD) for covariance estimation, further advancing research on decoding human speech imagery from EEG signals.

In summary, during SI-BCI neural decoding, optimizing the performance of traditional algorithms (e.g., CSP, RF) while further exploring deep learning-based approaches (e.g., CNN, RNN) has become a key trend in EEG signal preprocessing and feature extraction. Meanwhile, unconventional algorithms (e.g., covariance matrix-based methods) have also garnered increasing attention.

Datasets

We have compiled statistics on existing public and private SI-BCI datasets, and Fig. 8 visually presents their usage in percentage. Current research predominantly relies on private datasets from individual laboratories, with public dataset usage being relatively limited. This phenomenon is influenced by several factors, including the complexity of data collection, the need for data quality assurance, and limitations in participant numbers. Among these, the most frequently used public datasets are the Kara One, DaSalla, and ASU datasets. Thus, we provide an introduction to these three primary datasets, along with a recently released high-channel EEG dataset based on Chinese language stimuli.

Kara One (Zhao & Rudzicz, 2015): The Kara One dataset was collected and organized by the Department of Computer Science at the University of Toronto in collaboration with the Toronto Rehabilitation Institute. It integrates three modalities (EEG, facial tracking, and audio) during imagined and vocalized phonemes and single-word prompts, providing access to the brain’s language and speech production centers. Each participant was seated in a chair in front of a computer monitor. A Microsoft Kinect (v.1.8; Microsoft Corp., Redmond, WA, USA) camera was placed beside the screen to record the participant’s facial information and speech. Each trial consisted of four consecutive states:

1. A 5-s rest state, during which participants were instructed to relax and clear their minds.

2. A stimulus state, where a prompt text appeared on the screen, and its associated auditory utterance was played through computer speakers. This was followed by a 2-s period during which participants prepared their articulatory posture to begin vocalizing the prompt.

3. A 5-s imagined speech state, during which participants imagined speaking the prompt without actual movement.

4. A speaking state, where participants vocalized the prompt aloud. The Kinect sensor recorded audio and facial features during this phase.

ASU (Nguyen, Karavas & Artemiadis, 2017): The ASU dataset, provided by Arizona State University, is widely used for research and development in EEG-based BCI systems, particularly in identifying and decoding brain activity patterns related to imagined speech. A key feature of this dataset is that it includes EEG signals from multiple subjects performing specific imagined speech tasks. Participants were instructed to imagine hearing words or phrases rather than vocalizing them. The dataset comprises multiple EEG channels to capture brain activity. The task design incorporates various conditions, such as imagining different words or phrases, which aids researchers in exploring the patterns of brain activity during the process of speech imagination.

DaSalla (DaSalla et al., 2009): The DaSalla dataset was compiled and contributed by DaSalla and their team. This dataset generates speech-related potentials (SRP) in EEG signals for imagined vowels /a/ and /u/. Compared to other dataset’s imagined speech paradigms, SRP exhibits better variance. Each trial lasts approximately 1 min. It begins with a beep at 0 s to prepare the subject for the first step. The trial then proceeds through three stages:

1. A cross appears on the display for a random duration of either 2 or 3 s to prevent erroneous event-related potentials (ERPs) in subsequent steps.

2. Vowel imagination is performed in this step. It is represented by an open mouth outline, a rounded mouth, or a cross, indicating the /a/, /u/, or control state imagination tasks, respectively. This step lasts for 2 s. In this study, the control state is referred to as /no vowel/ since it involves no vowel. Each subject performed 150 trials, with 50 trials per task. The presentation of /a/, /u/, and /no vowel/ imagination tasks were randomized to train subjects to develop speech imagination tasks solely through endogenous stimuli.

3. A blank image is displayed on the screen for 3 s to indicate a rest period and the end of the trial, allowing the subject to wait for the next trial.

ChineseEEG (Mou et al., 2024): The ChineseEEG dataset is a high-channel EEG dataset based on Chinese language stimuli, providing a rich resource for semantic alignment and neural decoding. It includes high-density EEG data and synchronized eye-tracking data from 10 participants during approximately 13 h of silent reading of Chinese texts. The data is derived from the Chinese versions of two well-known novels, The Little Prince and Dream of the Wolf King. Each participant was exposed to diverse Chinese language stimuli, which is crucial for studying the long-term neural dynamics of language processing in the brain. Additionally, the 128-channel high-density EEG data offers superior spatial resolution, enabling precise localization of brain regions involved in language processing. With a sampling rate of 1 kHz, the dataset effectively captures the dynamic changes in neural representations during reading.

Machine learning based SI-BCI neural decoding

Most studies based on machine learning employ CSP for feature extraction and SVM for feature classification. The principle of SVM is to use mathematical optimization methods to find a hyperplane that maximizes the separation of two classes of data (Rezvani et al., 2024). The principle of CSP is to find a set of spatial filters that maximize the variance differences between different categories of EEG signals (Cai et al., 2024). This is the most common baseline approach. DaSalla et al. (2009), using the method of CSP combined with SVM, successfully achieved a classification accuracy rate of 68% to 78% for imagined pronunciations of the English vowels /a/ and /u/, laying the foundation for this field. Agarwal et al. (2020) combined CSP, statistical features (standard deviation, root mean square (RMS), energy), and wavelet transforms and employed a Random Forest classifier for multi-class classification of imagined vowels /a/, /u/, and the rest state. They achieved a maximum accuracy of 89% on data from three subjects, demonstrating the effectiveness of integrating CSP with statistical features in silent speech decoding. Rostami & Moradi (2015) proposed combining CSP with multiple bandpass filters and evidence theory, achieving a classification accuracy superior to traditional CSP methods. Table 3 shows the use of CSP and SVM along with their derived algorithms.

In addition to the aforementioned methods, several other feature classification approaches have also garnered significant attention. Idrees & Farooq (2016a) designed a vowel classification method based on wavelet decomposition and statistical features, achieving significant results with three subjects. Nguyen, Karavas & Artemiadis (2017) proposed a method based on Riemannian manifold features and RVM, achieving high accuracy in binary classification, but the accuracy in three-class classification significantly decreased. To improve the accuracy of three-class classification, they proposed a new method combining spatial covariance with an adaptive weighting model. Experimental results showed significant improvement, and surprisingly, subjects from the BCI blind group also achieved good results. Table 4 summarizes other machine learning feature extraction and classification algorithms.

Deep learning based SI-BCI neural decoding

There are numerous algorithms in deep learning, and within the field of SI-BCI, CNN, RNN, and capsule neural networks have garnered significant attention. CNNs have shown significant performance in image processing. They are deep, feedforward artificial neural networks. CNNs consist of an input layer, several hidden layers, and an output layer, with trainable weights and biases used to construct the neural framework. Each neuron receives data and then performs a dot product using a nonlinear function. The hidden layers include a series of convolutional layers and pooling layers, which multiply or perform dot product convolutions in other ways. Figure 9 is a typical CNN model generated using our visualization tool.

Research on CNNs tends to combine them with existing feature extraction methods, feeding the extracted features into the CNN to capture finer details. Saha & Fels (2019) proposed a novel hybrid deep neural network architecture for EEG speech imagery classification tasks. It is noteworthy that this was the first time CNNs were introduced into the field of speech imagery brain-computer interfaces, leading to groundbreaking advancements in subsequent research. Macías-Macías et al. (2020) first used CSP to extract signal features, then employed CNNs for feature classification. This combination effectively avoided the problem of CNNs losing significant features during the feature extraction process. Lee et al. (2021) used a CNN model to predict the word length corresponding to the input EEG signal and explored the impact of word length on speech imagery classification performance, concluding that training strategies based on limited word length could effectively improve overall classification accuracy. Park, Yeom & Sim (2021) used CNNs to classify brain spatial features formed by EEG channels and predict user states. However, this method was not compared with others, making it impossible to assess its relative performance. Ahn et al. (2022) modified the convolutional layers of CNNs by replacing single convolution kernels with multi-scale kernels, allowing for the extraction of features from different frequency bands and showing high and stable classification performance across different datasets. Kwon et al. (2023) proposed an improved deep learning model by incorporating multi-layer perceptrons (MLP) into ShallowConvNet and EEGNet to enhance cross-paradigm subject identification capabilities in BCI systems. Table 5 presents some applications of CNNs in SI-BCI neural decoding.

RNNs are used to process variable-length sequential data, such as time-series data and sound. They consist of a series of connected feedforward neural networks. The system uses temporal correlations to represent input history and predict outputs within the network. Figure 10 shows a typical recurrent neural network structure. Research on this type of network primarily focuses on the classification and prediction of continuously varying signals over time. Chengaiyan, Retnapandian & Anandan (2020) first used brain connectivity estimators and RNNs to identify vowels, achieving a classification accuracy of 72%, which demonstrated the effectiveness of this method. Retnapandian & Anandan (2023) proposed a method based on EEG subband signal features and the RNN model. They decomposed the EEG signal into five subband frequency bands and extracted energy coefficients (RMS, mean absolute value (MAV), integrated EEG (IEEG), simple square integral (SSI), variance of EEG (VAR), average amplitude change (AAC)) and relative power from each subband as features, using the RNN for multi-class vowel classification. After multiple rounds of experiments, the classification accuracy approached 90%. However, the training of the RNN uses the backpropagation through time (BPTT) algorithm. Due to difficulties with gradient explosion and vanishing, backpropagating gradients over long time intervals is challenging (Gomez et al., 2017; Vorontsov et al., 2017). Long short-term memory (LSTM) and gated recurrent units (GRU) have become popular alternatives to RNNs. Jeong et al. (2022) used GRUs to decode spatio-temporal frequency features at the sentence level. Hernandez-Galvan, Ramirez-Alonso & Ramirez-Quintana (2023) employed 1D convolution layers and two layers of bidirectional GRUs to extract time-frequency features from EEG signals, using prototypical networks for classification. The average accuracy across two public datasets was above 91%. Table 6 presents the applications of RNNs and their variants.

CapsNets aim to address the challenges faced by CNNs in handling spatial hierarchies and pose variations in images. Its core advantage lies in its ability to capture the hierarchical structure and pose variations of objects within images, providing a more refined and robust feature representation than CNNs. Figure 11 shows a simple capsule network model (Aboussaleh et al., 2024). Macías-Macías et al. (2023) proposed a capsule network-based classification model, CapsK-SI, which achieved recognition of bilabial sounds, nasal sounds, consonant-vowel combinations, and the /iy/ and /uw/ vowels. It also generated relevant brain activity maps, offering clues for further understanding the neural mechanisms of speech imagination. Ramirez-Quintana et al. (2023) proposed a deep capsule network model called CapsVI, which achieved an average accuracy of 93.32% in paired classification. Table 7 presents applications of CapsNets and its variants.

Research on other deep learning methods has also demonstrated their effectiveness in decoding SI-BCI systems. Saji et al. (2020) and Watanabe et al. (2020) respectively employed DWT and speech amplitude envelope rhythm extraction to extract EEG features, using different classifiers for speech imagination classification, demonstrating the application potential of deep learning models in different linguistic contexts. Islam & Shuvo (2019) proposed a deep learning-based improvement method for beta-band selection, combining DenseNet and Gramian Angular Field to address the issue of EEG signal classification accuracy, significantly improving classification accuracy. Panachakel & Ganesan (2021) sed a transfer learning model based on ResNet50 to classify the ASU imagined speech EEG dataset, showcasing the effectiveness of transfer learning in EEG signal classification. Table 8 presents the applications of some other deep learning algorithms.

Algorithm comparison and evaluation

The compilation and analysis of existing SI-BCI neural decoding algorithms are summarized in Fig. 12. The following trends emerge from a quantitative analysis: Machine learning methods predominantly utilize SVM, which demonstrate stable performance and high classification accuracy, making them the baseline method in this field and accounting for the majority of applications. Among deep learning algorithms, CNNs and their variants have demonstrated exceptional performance.

Model performance evaluation involves various metrics, such as accuracy, precision, recall, F1-score, confusion matrix, receiver operating characteristic (ROC) curves, and area under the curve (AUC). Each metric emphasizes different aspects of model performance, making the selection of appropriate evaluation criteria critical. In the field of SI-BCI, average accuracy is the primary performance metric for evaluating SI-BCI algorithms. Additionally, metrics such as recall, precision, and F1-score are often included to offer a more comprehensive assessment of model performance. Thus, accuracy is chosen as the main evaluation metric in this study to facilitate clear comparison of the relative performance of various models. For both binary and multi-class classification tasks, accuracy is defined as the proportion of correctly classified samples to the total number of samples. However, in multi-class classification, accuracy calculations must account for multiple categories, which makes comparisons between binary and multi-class tasks more nuanced. Table 9 presents the highest accuracy achieved by different models and methods.

Difficulty in acquiring high-quality signals and the principles of brain region connectivity in speech processing

Challenges: Our analysis of Tables 1–8 indicates that most studies use EEG acquisition devices with 32 or more channels. However, signal quality is often influenced by participant characteristics, including age, gender, and handedness. Furthermore, prolonged experimental sessions may lead to participant fatigue, impacting attention and reaction speed, which increases EEG variability and noise artifacts (e.g., ocular and electromyographic artifacts), ultimately reducing signal quality (Zhang, Li & Chen, 2020; Borirakarawin & Punsawad, 2023; Kim, Lee & Lee, 2020; Lee et al., 2021). Additionally, the experimental environment plays a crucial role. Prolonged recording sessions increase the risk of environmental interference during signal acquisition. For instance, Jafferson et al. (2021) reported an 18% error rate in classification results due to electrical interference during data collection. In “SI-BCISI-BCI Paradigm”, we reviewed recent advancements in brain connectivity studies. Broca’s and Wernicke’s areas are known to be essential in speech processing—Broca’s area is responsible for speech production and articulation, whereas Wernicke’s area facilitates speech comprehension. However, many existing studies collect EEG signals from the entire brain without focusing on electrode placement in these critical speech-related regions (Wang et al., 2021; Sikdar, Roy & Mahadevappa, 2018; Zhao, Liu & Gao, 2021; Pan et al., 2024; Wang et al., 2020).

Future Directions: To advance the understanding of speech-related brain mechanisms, the following research directions are proposed: Interdisciplinary research–Strengthen collaboration with fields such as anatomy and neuroscience to conduct more precise structural and functional analyses of speech-related brain regions, particularly Broca’s and Wernicke’s areas. This will enable deeper investigation into the neural signal transmission and cortical activation mechanisms underlying speech intention, potentially uncovering functional micro-units within these key linguistic centers (Zhang, Guo & Chen, 2023). Optimization of EEG acquisition equipment–Design and refine more efficient and sensitive EEG acquisition devices capable of capturing a broader range of neural activity frequencies. Enhancing the precision of EEG recordings will improve the detection of subtle neural activities involved in speech generation. These research directions hold promise for revealing the high-level cognitive functions underlying speech generation and comprehension and advancing the development of high-sensitivity EEG systems for applications in neurolinguistics.

Innovation in SI-BCI paradigms: limited focus on words and sentences

Challenges: As summarized in “SI-BCI Neural Encoding” and Tables 1–8, vowel imagery has been extensively studied (Chengaiyan & Anandan, 2022; DaSalla et al., 2009; Cooney, Folli & Coyle, 2018; Wu & Chen, 2020; Agarwal et al., 2020), whereas research on short words, long words, and phrases—such as English words and Chinese characters—is relatively scarce (Alizadeh & Omranpour, 2023; Kamble, Ghare & Kumar, 2022; Sereshkeh et al., 2017; Panachakel & Ramakrishnan, 2022). This limitation may stem from the design of existing paradigms, as a single paradigm may not be sufficient to fully address complex speech imagery tasks.

Future Directions: To innovate SI-BCI paradigms, future research should focus on designing paradigms that incorporate word-, character-, phrase-, and sentence-level speech intention tasks. Additionally, integrating multimodal signals (e.g., electromyography (EMG), eye movement signals) and multimodal paradigms (e.g., motor imagery and external stimuli) could improve the naturalness and usability of speech intention paradigms for real-world applications. Wang et al. (2019, 2022) proposed a hybrid BCI concept by designing a three-class task system that combines motor imagery and speech imagery, leading to improved classification accuracy. This suggests that multimodal data fusion can significantly enhance the efficiency and precision of BCI systems. Tong et al. (2023) further explored multimodal speech imagery applications by integrating silent reading and handwriting imagery, which elicited stronger EEG features compared to traditional motor imagery paradigms, thereby improving classification accuracy. Naebi & Feng (2023) proposed a novel communication imagery model as an alternative to speech-dependent mental tasks. They introduced a lip imagery model, achieving classification results ranging from 55% to 98%, with the highest performance achieved through a novel signal combination approach. Notably, Silva et al. (2024) leveraged BCI and natural language processing (NLP) technologies to enable a patient to communicate in Spanish and English. This groundbreaking study may provide new insights into the development of bilingual BCIs.

Excessive signal preprocessing and lack of multimodal data fusion in SI-BCI neural encoding

Challenges: Some studies have applied Autoencoder layers to handle incomplete and noisy EEG signals (Zhang et al., 2018; Ali, Mumtaz & Maqsood, 2023), while others have employed noise-assisted multivariate empirical mode decomposition (NA-MEMD) for signal processing (Park & Lee, 2023). However, these methods significantly increase preprocessing time, hindering real-time applications. He et al. (2023) highlighted that relying solely on EEG signals for unimodal data processing limits the model’s ability to capture the complexity of brain activity.

Future Directions: Beyond EEG, other physiological signals—such as fNIRS, electrooculography (EOG), and EMG—should be integrated into SI-BCI neural encoding for imagined speech decoding. Although most research focuses on single-modal signals, multimodal classification has emerged as a promising direction (Wang et al., 2023). Guo & Chen (2022) developed an fNIRS-based 4-class speech imagery BCI, achieving >70% mean subject-specific accuracy in binary-class settings (all pairwise comparisons between two vowels) across 0–2.5 s and 0–10 s time windows. Vorreuther et al. (2023) and Rezazadeh Sereshkeh et al. (2019) proposed a multi-class hybrid fNIRS-EEG BCI for imagined speech, demonstrating significantly higher accuracy than unimodal approaches. These findings highlight the complementary nature of diverse neural signals in enhancing BCI performance. Future studies should adopt mobile, wearable sensors to collect multimodal physiological data (facial expressions, EEG, etc.) and fuse them within a unified framework to improve online real-time feature extraction and imagined speech recognition accuracy.

Limited SI-BCI datasets and overreliance on binary classification tasks

Challenges: In Tables 1–8 and “Difficulty in Acquiring High-Quality Signals and the Principles of Brain Region Connectivity in Speech Processing”, we analyzed available datasets, revealing that most studies rely on private datasets with fewer than ten participants. This poses a significant challenge for neural decoding, as the lack of large datasets prevents deep learning models from leveraging large-scale data analysis, thus affecting model performance. Additionally, most studies still focus on binary classification tasks, limiting their applicability to real-world BCI applications. Furthermore, the absence of diverse evaluation metrics hinders comprehensive performance assessment.

Future Directions: In SI-BCI neural decoding, it is crucial to continue advancing deep learning algorithms, particularly CNN-based models, to enhance the capability of handling complex, nonlinear transformations. Expanding research from binary classification to multiclass classification is essential. For example, Hernandez-Galvan, Ramirez-Alonso & Ramirez-Quintana (2023) achieved high accuracy across multiple classes using two public datasets, demonstrating the feasibility of multiclass SI-BCI tasks. Current public datasets are predominantly vowel-based, such as Kara One and AUS. Therefore, developing high-quality word- and sentence-level public datasets is a critical direction for future SI-BCI advancements. A recent contribution in this area is ChineseEEG, a high-density EEG dataset compiled by Mou et al. (2024), containing approximately 13 h of Chinese text reading EEG recordings from 10 participants. This dataset fills a significant gap in Chinese language EEG datasets and represents a major step toward improving linguistic diversity in SI-BCI research.