Feature selection for emotion recognition in speech: a comparative study of filter and wrapper methods

TESS

The TESS dataset includes recordings of two English actresses, one aged 26 and the other 64. The audio recordings are two seconds in length featuring anger, disgust, fear, happiness, neutrality, surprise, and sadness. The dataset includes 2,800 audio files, with 400 files allocated to each emotion category (Pichora-Fuller & Dupuis, 2020; University of Toronto, 2025).

CREMA-D

The CREMA-D dataset includes 7,442 unique audio samples from 91 actors of different racial and ethnic backgrounds (Science, 2025). The dataset comprises 48 male actors and 43 female actors, each uttering 12 sentences. The audio recordings are two seconds in length and express six different emotions: anger, happiness, neutrality, disgust, sadness, and fear (Cao et al., 2014).

Table 2 reflects the final distribution of emotion samples used in this study before augmentation. The TESS dataset contributes an even 400 samples per emotion for seven categories, excluding “calm” (Pichora-Fuller & Dupuis, 2020; University of Toronto, 2025). CREMA-D contains varied samples across six emotions, excluding “calm” and “surprise” (Cao et al., 2014), while RAVDESS provides all eight emotions including the only instances of “calm” and “surprise” (Livingstone & Russo, 2018). These distributions collectively form the merged dataset shown in Fig. 2.

RAVDESS

The RAVDESS dataset comprises 24 professional actors, 12 male and 12 female, vocalizing two lexically matched statements in a neutral North American accent (Livingstone & Russo, 2018). The dataset includes seven emotions: “calm”, “happy”, “sad” “surprise”, “angry”, “disgust”, and “fearful”. Each emotion is presented at two levels of intensity- (normal, strong) and a neutral expression. The dataset includes audio recordings in three modalities: audio-only, audio-video, and video-only. The dataset contains 2,076 audio recordings with an average length of three seconds per recording (Livingstone & Russo, 2018).

Data analysis

To create a unified dataset for analysis, all audio recordings from the three datasets (TESS, CREMA-D, and RAVDESS) were standardized. Specifically, each audio clip was resampled to a common sampling rate of 16 kHz to ensure consistency. Audio lengths were normalized to 2 s by trimming or zero-padding as needed. Metadata was harmonized, and a consolidated dataset was formed for feature extraction and model training. The statistics of the resulting combined dataset are illustrated in Fig. 2.

Figure 3 presents a comparative visual analysis of audio features for “happy” and “sad” emotional states, depicted through waveplots and spectrograms at different scales. The waveplot of an audio file visually represents the audio signal’s amplitude variations over time. This plot is typically used to display the structure of an audio track at a glance, showing where the sound is loud or quiet, the start and end of different sections, and where silence occurs within the file. The waveplot of the audio data is represented in Figs. 3A and 3D for the “happy” and “sad” emotion categories, respectively. These waveplots show notable differences in the amplitude envelope of the audio signals. The waveform associated with happiness demonstrates a greater amplitude variability, which may indicate a more dynamic and expressive speech pattern commonly found in joyful communication. In contrast, the waveform for sadness displays a flatter envelope, suggesting a more monotone and subdued vocal expression.

Figures 3B and 3E represent the short-time Fourier transform (STFT) of the happy and sad emotions on a linear scale. These figures reveal the frequency content of the happy and sad emotions on a linear scale. The happy emotion spectrogram has a broader spread of energy across the frequency spectrum, particularly noticeable in the mid to high-frequency range, which corresponds to a more energetic articulation. This contrast is stark against the sad emotion spectrogram, where the energy is more confined and less varied, reflecting the typically lower and narrower pitch range observed in sad speech.

The STFT is an analysis technique used to determine the sinusoidal frequency and phase content of local sections of a signal as it changes over time. This is particularly useful for audio signals, where we must analyze how frequencies vary with time. STFT works by dividing the signal into shorter segments of equal length and computing the Fourier transform separately on each segment. This results in a two-dimensional representation of the signal, where one dimension is time and the other is frequency, with the amplitude given by the magnitude of the transform (Kehtarnavaz, 2008). The STFT of a speech sample can be represented on either a linear or logarithmic scale. Each frequency bin is spaced equally across the frequency axis on a linear scale. This type of scaling is straightforward and gives a uniform spectrum view, which is good for general purposes, particularly where higher frequency resolution is needed across the spectrum. In a logarithmic scale, each octave (doubling of frequency) has the same space on the axis, which means higher frequencies are compressed, and lower frequencies are expanded in the visual representation. This scaling is more aligned with how humans perceive sound, where we tend to notice differences in lower frequencies more than in higher ones (Krishnan, 2021).

Figures 3C and 3F represent these emotions on a log scale, and allow for a deeper inspection of the lower frequencies, which are crucial for identifying speech formants. The happy emotion spectrogram on a log scale shows a richer harmonic structure, with clearly defined formants that suggest a greater degree of vocal tract resonance. In comparison, the sad emotion spectrogram displays less pronounced formants, implying a reduction in resonance which can be attributed to a more closed or constricted vocal tract during sad expressions.

Adding noise

To add noise to the audio data, we defined a function that receives the audio array as input and adds random noise before returning a noisy data array as output. A random factor of between 0 and 0.035 is used. The noise amplitude is directly proportional to the random factor and the maximum value of the data. A noisy audio data is represented in Fig. 4B.

Stretching

The stretch function was designed to either elongate or condense an array of audio data temporally, dependent on a specified rate factor. When the rate factor is set below 1, the audio is elongated, resulting in a slower playback. Conversely, a rate factor above 1 will condense the audio, increasing the playback speed. In our specific application, we have employed a rate factor of 0.7 to slow down the audio data. Figure 4C represents the stretched audio sample.

Pitch

To alter the pitch of the audio data, we defined a function that takes in the audio array, a pitch factor, and the sampling rate as input and alters the frequency or pitch of the data. The pitch factor determines how much the pitch should be altered. When this value is above 1, it implies that the pitch is increased; when it is below 1, it is decreased. The output is an audio data array, pitch-shifted by a pitch factor of 0.7. Figure 4D represents audio sample with pitch-shift.

Shifting

The shift function, on the other hand, is responsible for temporarily displacing an audio data array by a randomly determined interval. This interval is the product of a random integer, selected from the range of −5 to 5, and the factor 1,000. The displacement is achieved through the numpy.roll function, which cyclically shifts the elements of the array. The outcome of this function is the time-shifted audio data array represented in Fig. 4E.

In the data augmentation phase, two augmentation techniques were applied per sample, resulting in three total variants per instance (original + two augmentations). This tripled the dataset size from 11,682 to 35,046 samples. The original combined dataset contained varying numbers of samples per emotion category, as shown in Fig. 2. We retained all eight emotion categories: fear, happy, sad, neutral, angry, disgust, surprise, and calm. Notably, calm and surprise were underrepresented with 192 and 592 samples, respectively. Table 3 presents the number of samples before and after augmentation for each emotion.

Filter method

Filter methods are employed to assess the relevance of features based on statistical measures, providing scores for each feature. These scores are used to rank features, selecting those most likely to be useful for model training or removing those that do not meet a predefined relevance threshold. This approach is especially efficient as it is applied as a preprocessing step, independent of the model training phase.

Correlation-based (CB) feature selection: CB feature selection is particularly effective in the context of emotion recognition as it identifies features that show a strong correlation with the target emotional states but minimal correlation with each other. This helps reduce multicollinearity and improve the generalizability of the model.

The effectiveness of a feature set can be evaluated using a heuristic that considers both the individual predictive ability of each feature and the redundancy among them. The merit of a feature subset S containing k features can be computed as:

(10) $$M_s=\frac{k\overline{r_{cf}}}{\sqrt{k+k(k-1)\overline{r_{ff}}}}$$where $\overline{r_{cf}}$ is the average correlation of the features with the class, and $\overline{r_{ff}}$ is the average inter-correlation among the features.

Mutual information (MI): MI is used to quantify how much information one feature shares with the target variable, making it an invaluable tool in determining feature relevance without assuming linear relationships. For emotion recognition, MI can highlight non-linear dependencies that are vital for understanding complex emotional patterns.

The mutual information between a feature X and the target variable Y is calculated as:

(11) $$I(X;Y)=\sum _{x\in X}\sum _{y\in Y}p(x,y)\log \left(\frac{p(x,y)}{(y)}\right)$$where $p(x,y)$ is the joint probability distribution of X and Y, and p(x) and p(y) are the marginal distributions.

Wrapper method

Wrapper methods involve a search for the optimal subset of features that maximizes a specific model’s performance criterion. Unlike filter methods, wrappers evaluate features based on the model’s accuracy, making them more computationally intensive but potentially more accurate.

Recursive feature elimination (RFE): RFE is well-suited for emotion recognition as it iteratively refines the feature set to retain only those that contribute most significantly to model performance. By integrating RFE, emotion recognition models can focus on the most impactful features, potentially enhancing both accuracy and efficiency.

RFE operates through the following steps:

1. Initialize the model on the full set of features and assess their importance.

2. Eliminate the least important feature(s) based on the model’s criteria.

3. Re-train the model on the reduced set of features.

4. Repeat the process until a predetermined number of features is left.

Employing these feature selection strategies helps streamline the modeling process, particularly in complex datasets typical of emotion recognition tasks. By focusing on the most relevant features, these methods can reduce overfitting and improve the predictive power of the models.

System configuration

The experimental system is configured with a high-performance 64-bit architecture designed for intensive data processing. It features symmetric multiprocessing (SMP) and vsyscall32 capabilities. The system includes 130 GB of system memory, crucial for managing large datasets and complex algorithms efficiently, preventing performance bottlenecks. The central processing unit (CPU) is an Intel® Core™ i9-10940X operating at 3.30 GHz with a capability to reach up to 4,800 MHz. This robust CPU setup supports high-speed processing and multitasking, making it highly suitable for demanding machine learning and data analysis tasks.

Implementation details

The dataset was split using a random shuffling strategy applied to individual audio samples, and by a proportion of 70:15:15 for the train, test, and validation sets, respectively. We designed and trained a CNN model structured using a sequence of convolutional and pooling layers. The first layer is a 1D convolution layer with 256 filters, a kernel size of 5, a stride of 1, and ‘same’ padding. It uses the rectified linear unit (ReLU) as its activation function. A max-pooling layer follows the first convolution layer, with a pool size of 5 and a stride of 2, also with ‘same’ padding. These layers repeat the structure of a Conv1D followed by a MaxPooling1D but with different numbers of filters. The second set also has 256 filters, followed by a set with 128 filters and then 64 filters. This configuration allows the network to capture hierarchies of features at different levels of abstraction. Dropout layers are interspersed after some of the pooling layers, with dropout rates of 0.2 and 0.3. A flatten layer flattens the output from the previous convolutional and pooling layers to form a single long feature vector needed for the dense layers that follow. A fully connected layer with 32 units and ReLU activation function is used to interpret the features extracted by the convolutions and pooling. The final output layer has 8 units with a softmax activation function. The model uses the Adam optimizer, a popular choice for deep learning applications. The loss function is categorical cross-entropy.

Evaluation metrics

Several evaluation metrics were utilized to assess the effectiveness of the emotion recognition solutions developed in this study. These metrics provide insights into different facets of model performance, from overall accuracy to the balance between precision and recall. Below, we detail each of these metrics.

• Accuracy (ACC) is one of the most intuitive performance measures (Bown, 2024). It is defined as the ratio of correctly predicted observations to the total observations: (12) $$ACC=\frac{TP+TN}{TP+TN+FP+FN}$$where TP, TN, FP, and FN represent the true positives, true negatives, false positives, and false negatives, respectively. High accuracy indicates that the model performs well across all classes. However, accuracy alone can be misleading, especially in datasets with imbalanced classes.

• Precision (P), or positive predictive value, measures the accuracy of positive predictions (Foody, 2023). Formally, it is defined as the ratio of correctly predicted positive observations to the total predicted positives: (13) $$P=\frac{TP}{TP+FP}$$High precision indicates a low rate of false positives, which is crucial in applications where the cost of a false positive is high.

• Recall (R), or sensitivity, measures the ability of a model to find all the relevant cases within a dataset (Foody, 2023). It is defined as the ratio of correctly predicted positive observations to all observations in the actual class. (14) $$R=\frac{TP}{TP+FN}$$High recall is particularly important in situations where missing a positive instance is significantly detrimental.

• The F1-score (F1) is the weighted average of precision and recall. Therefore, this score takes both false positives and false negatives into account (Foody, 2023). It is especially useful when the class distribution is uneven. The F1-score is defined as: (15) $$F1=2\cdot \frac{P\cdot R}{P+R}$$The F1-score is a better measure to use when seeking a balance between precision and recall, and there is an uneven class distribution.

• Support is the number of actual occurrences of the class in the specified dataset. While not a performance measure in itself, support provides valuable context when comparing the quality of classification reports across different classes (Foody, 2023). It is crucial for assessing the reliability of the metrics above, particularly in datasets like ours, where some classes (e.g., calm and surprise) are significantly underrepresented.

Correlation-based method

The following thresholds were used to evaluate the CB method: 0.02, 0.05, 0.1, and 0.2. These thresholds helped determine the number of features to be selected, based on the CB algorithm. Figures 5A, 5B, and 5C represent the confusion matrix of the emotion recognition result based on the CB method for thresholds 0.05, 0.02, and 0.1, respectively. The best result was obtained when the threshold was set to 0.02, resulting in a selection of the best $158$ features and an accuracy of $60.45\mathrm{\%}$.

Table 4 provides more insight into the precision (P), recall (R), and F1-score for the different thresholds. From observation, the performance varies across different thresholds for each emotion category, indicating that the choice of threshold, significantly impacts the performance of the feature extraction method. Different emotion categories exhibit varying performance across the thresholds. For example, the “Surprise” category generally achieves higher precision, recall, and F1-score compared to other categories across all thresholds. In some cases, adjusting the threshold leads to improvements or deteriorations in performance metrics. For instance, for the “Angry” category, lowering the threshold from 0.1 to 0.02 results in an increase in precision and F1-score, albeit a slight decrease in recall. The performance of emotion categories like “Neutral” and “Sad” tends to be lower compared to others, which might indicate class imbalance issues or inherent difficulty in distinguishing these emotions from speech features alone.

One more observation is that lower correlation thresholds (e.g., 0.02) tend to result in a broader selection of features, potentially capturing more diverse information but also introducing more noise. Higher thresholds (e.g., 0.1) are more conservative, selecting fewer features but potentially more relevant ones. Also, the sensitivity of performance to changes in the correlation threshold varies across emotions. For example, “Surprise” appears to be less sensitive, maintaining relatively high performance across different thresholds, while “Calm” exhibits more variability.

In this approach, the best performance was obtained at a threshold of 0.02 on the “Surprise” emotion, with a precision score of 0.88, a recall of 0.90, and an F1-score of 0.89. The poorest performance was observed at a threshold of 0.05 on the “Disgust” emotion with a precision score of 0.50, a recall of 0.49, and an F1-score of 0.49.

Mutual information feature extraction method

In implementing the mutual information approach, different values of a ‘K’ factor need to be defined. In this study, K represents the number of features selected from the full feature set based on their mutual information scores with the emotion labels. Five different values were employed: 50, 80, 100, 120, and 150. Figure 6 represents the confusion matrix for these various values of K, with the best performance occurring at K = 120, with a test accuracy of $60.53\mathrm{\%}$, and $120$ features.

The results of the confusion matrices are elaborated in Table 5, and shows various observations. Firstly, different emotion categories exhibit varying performance across different values of K. For example, the “Surprise” category generally achieves higher precision, recall, and F1-score compared to other categories across all values of K. Secondly, for each emotion category, there seems to be an optimal value of K that maximizes the performance metrics. This optimal value may vary depending on the specific characteristics of the emotion category and the dataset. Also, adjusting the value of K can lead to trade-offs between precision and recall. For instance, increasing K may lead to higher recall but lower precision, and vice versa.

Once again, emotion categories like “Calm” and “Sad” exhibit lower performance compared to others, which might be attributed to class imbalance issues or inherent difficulty in distinguishing these emotions from speech features alone.

In this approach, the best performance was obtained at a K value of 150 on the “Surprise” emotion, with a precision score of 0.80, a recall of 0.89, and an F1-score of 0.84. The poorest performance was observed at a K value of 50 on the ‘Calm’ emotion with a precision score of 0.39, a recall of 0.27, and an F1-score of 0.32.