Leveraging PSO-MLP for intelligent assessment of student learning in remote environments: a multimodal approach

Motivation for the study and methodology

The increasing prevalence of remote and mobile learning environments has introduced significant challenges in accurately assessing student engagement, emotional well-being, and learning effectiveness. Traditional evaluation methods, relying heavily on self-reporting or single-modal behavioral indicators, are often limited by subjectivity, latency, and low adaptability. This study is motivated by the urgent need to develop intelligent, real-time, and objective evaluation tools that can adapt to the multimodal and dynamic nature of distance education. The decision to integrate multimodal physiological signals—such as electroencephalography (EEG), galvanic skin response (GSR) and heart rate—with intelligent classification models is based on the understanding that physiological data serve as reliable proxies for emotional and cognitive states, which are essential for effective learning.

To address the inherent complexity and non-linearity of such data, a multilayer perceptron (MLP) was selected due to its strong capacity for modeling high-dimensional feature interactions. However, MLP’s performance is highly sensitive to its initial hyperparameters and susceptible to local minima during training. Therefore, particle swarm optimization (PSO) was introduced as a metaheuristic method to efficiently optimize MLP parameters. PSO offers global search capabilities and faster convergence, making it well-suited for optimizing neural networks in scenarios involving heterogeneous, high-noise inputs. The combined PSO-MLP framework thus aligns with the study’s motivation to create a robust, adaptive, and interpretable model for intelligent assessment in real-world, data-rich educational settings.

Therefore, the contributions of the article are:

(1) Integration of multimodal data for enhanced student evaluation: The article introduces an intelligent evaluation method that integrates pre-class mental status surveys with big health data from both teachers and students. This multimodal approach enables a more comprehensive understanding of students’ learning states and classroom enthusiasm, representing a significant improvement over existing deep learning models that struggle with remote education data.

(2) Application of particle swarm optimization (PSO) to improve model performance: By applying the PSO model to optimize the Multilayer Perceptron (MLP), the article enhances the efficiency and accuracy of the learning state assessment. The PSO-MLP model achieves a notable accuracy of 0.891, demonstrating improved performance over traditional methods and offering a technical solution for better recognizing and managing students’ learning conditions.

‘Related Works’ analyzes the application status of the PSO-MLP model under multimodal data, and ‘Materials and Methods’ expounds on the implementation process of the intelligent evaluation method of students’ learning status based on multimodal data. In ‘Experiments and Analysis’, the robustness of the model and the effect of sentiment classification are experimented with. ‘Discussion’ concludes with a summary and limitation analysis.

Data preprocessing steps

The accuracy and robustness of multimodal learning state recognition heavily depend on the quality and consistency of the input data. Therefore, an extensive data preprocessing pipeline was implemented to standardize the input from various physiological signal sources, including EEG, GSR, heart rate, and facial expression. The preprocessing procedures were designed to ensure data integrity, temporal alignment, and feature relevance, thereby facilitating effective learning by the PSO-MLP model.

Remote monitoring

The underlying distribution of real data samples is captured through the GAN generator and new data is generated, which divides the original data into three groups according to the labels (high “2”, medium “1”, and low “0”). The objective function of GAN is expressed as follows.

(1) $$\underset{G}{\mathrm{m}\mathrm{i}\mathrm{n}}\underset{D}{\mathrm{m}\mathrm{a}\mathrm{x}}\left\{f\left(D,G\right)=E_{x\sim \mathrm{P}\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\left(x\right)}\left[\mathrm{l}\mathrm{o}\mathrm{g}D\left(x\right)\right]+E_{z\sim Pz\left(z\right)}\left[1-\mathrm{D}\left(G\left(z\right)\right)\right]\right\}$$where x fits the true data distribution ${\mathrm{P}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}\left(\mathrm{x}\right)$, z fits the prior distribution ${\mathrm{P}}_{\mathrm{z}}\left(\mathrm{z}\right)$, and $\mathrm{E}(\cdot )$ represents the expected value. The Discriminator D is a neural network designed to distinguish between real data samples and those generated by the Generator. The Generator G is a neural network that generates synthetic data samples from a noise vector z sampled from a prior distribution ${\mathrm{P}}_{\mathrm{z}}\left(\mathrm{z}\right)$.

Therefore, the Wasserstein distance is introduced based on GAN, which effectively addresses the issue that the distance cannot be reflected when the two distributions have no overlap. The Wasserstein distance is expressed as follows.

(2) $$\mathrm{W}\left(\mathrm{p},\mathrm{q}\right)=\underset{\gamma \sim \left(p,q\right)}{\mathrm{i}\mathrm{n}\mathrm{f}}{E}_{\left(x,y\right)\sim \gamma }\left[\Vert x-y\Vert \right]$$where $\mathrm{i}\mathrm{n}\mathrm{f}\{\cdot \}$ set lower bounds, $\prod \left(\mathrm{p},\mathrm{q}\right)$ is the distribution of all possible sets of joint distribution, joint distribution, for each possible $\mathrm{\gamma }\sim \prod \left(\mathrm{p},\mathrm{q}\right)$, calculate the expectation of distance $\Vert \mathrm{x}-\mathrm{y}\Vert$, called ${\mathrm{E}}_{\left(\mathrm{x},\mathrm{y}\right)\sim \mathrm{\gamma }}\left[\Vert \mathrm{x}-\mathrm{y}\Vert \right]$. The lower bound on the expectation is the Wasserstein distance of the distributions p and q.

After the aforementioned preprocessing steps, we obtained the processed physiological signal S(t). Figure 1 demonstrates the preprocessing effects: Fig. 1A shows the raw data during the emotional stimulation phase; Fig. 1B displays the data after linear interpolation; Fig. 1C presents the data after noise reduction with a filter; Fig. 1D shows the normalized data; Fig. 1E depicts the data after processing with GAN. These techniques not only enhance the accuracy of emotion recognition but also provide crucial technical support and data foundation for applications in remote monitoring and education.

MLP-based health condition recognition

MLP is the most classical neural network, also known as a feedforward neural network or artificial neural network. The traditional perceptron is a linear model that can only handle simple binary classifications and performs poorly in nonlinear problems. Then, by adding hidden layers and activation functions, it effectively improves the nonlinear expression ability and is widely used in data mining, machine learning, and other fields (Xiong et al., 2021). The specific structure of the MLP is shown in Fig. 2.

For a multilayer perceptron, its internal neural network layer is also divided into three types: input layer, hidden layer, and output layer. Corresponding weights connect different layers. Input $\mathrm{u}={\left[{\mathrm{u}}_1,{\mathrm{u}}_2,\cdots ,{\mathrm{u}}_{\mathrm{n}}\right]}^{\mathrm{T}}$ and $\mathrm{y}=$ ${\left[{\mathrm{y}}_1,{\mathrm{y}}_2,\cdots ,{\mathrm{y}}_{\mathrm{p}}\right]}^{\mathrm{T}}$ are mapped by the weight w shown in Formula (3) between each layer:

(3) $$w={\left[{\left(w^1\right)}^{\mathrm{T}},{\left(w^2\right)}^{\mathrm{T}}\right]}^{\mathrm{T}}.$$

Once the MLP is constructed, the number of parameters can be calculated. The learning procedure for the MLP involves estimating the parameters. To optimize the MLP training, a general state-space model can be expressed by Formulas (4) and (5):

(4) $$w_k=w_{k-1}+v_k$$

(5) $$y_k=g\left(w_k,u_k\right)+{\omega }_k.$$

For its learning process and parameter iteration, it can be calculated through the forward and backward propagation algorithm. The forward propagation algorithm is used to establish the input, output, and network connection modes, as well as to determine the number of layers and the corresponding nodes in the network. The forward propagation algorithm only initializes the neural network and does not adjust the weights and offsets. It only performs feedforward calculations and cannot solve problems. Therefore, we need to use the backpropagation method to adjust the internal parameters of the network and optimize its internal structure to minimize the defined loss function. The specific process of the reverse neural network is shown in Algorithm 1.

This algorithm represents the foundational process of training a backpropagation neural network, wherein the minimization of the loss function guides the iterative adjustment of weights and biases. The forward propagation computes activations, while backpropagation calculates the gradients, enabling the optimization of network parameters. The process iterates until the changes in parameters fall below a predefined threshold, ensuring convergence to an optimal solution.

PSO-based model optimization

The neural network methods typically employ gradient descent for learning and optimizing network weights during model optimization, making them sensitive to initial values. Improper selection of initial values can cause optimal weights and biases to converge to local extreme points, significantly reducing model accuracy. Therefore, this study aims to utilize the PSO method to optimize the initial values of MLP, thereby enhancing performance. This article evaluates the performance of particles through their adoption in the study.

The particles in the particle swarm used for the MLP parameters optimization are defined as follows:

(6) $$P_j=\lfloor C_j{\epsilon }_j{\sigma }_j\rfloor j=1,2,\dots ,Q$$where Q is the particle number total.

First, initialize the particles randomly and then update them iteratively. Three characters define each particle in iteration k.

• (1)

  Location in search space $P_j\left(k\right)$

• (2)

  The best position $P_{jbest}\left(k\right)$ at k iteration

• (3)

  Flight speed $V_j\left(k\right)$

In addition, global optimal position of the entire particle swarm is defined as $P_{jbest}\left(k\right)$, so the function of velocity $V_j$ and position $P_j$ updated iteratively by each particle during flight is defined as:

(7) $$\alpha \left(k\right)=\left({\alpha }_{max}-{\alpha }_{min}\right)(k/K{)}^2+{\alpha }_{min}$$

(8) $${\mathrm{c}}_1\left(\mathrm{k}\right)=\left({\mathrm{c}}_{1\mathrm{m}\mathrm{a}\mathrm{x}}-{\mathrm{c}}_{1\mathrm{m}\mathrm{i}\mathrm{n}}\right)(\mathrm{k}/\mathrm{K}{)}^2+{\mathrm{c}}_{1\mathrm{m}\mathrm{i}\mathrm{n}}$$

(9) $$c_2\left(k\right)=\left(c_{2\mathrm{m}\mathrm{i}\mathrm{n}}-c_{2\mathrm{m}\mathrm{a}\mathrm{x}}\right)(k/K{)}^2+c_{2\mathrm{m}\mathrm{a}\mathrm{x}}$$

(10) $$v\left(k+1\right)=\alpha \left(k\right)v_j\left(k\right)+c_1\left(k\right)r_1\lfloor P_j\left(k\right)-P_{jbest}\left(k\right)\rfloor +c_2\left(k\right)r_2\lfloor P_j\left(k\right)-P_{gbest}\left(k\right)\rfloor$$

(11) $$P_j\left(k+1\right)=P_j\left(k\right)+v_j\left(k+1\right)$$

In Formula (7) to (11), $\alpha \left(k\right)$ is the inertial velocity weight, $c_1\left(k\right)$ and $c_2\left(k\right)$ is the coefficient of acceleration, $r_1$ and $r_2$ are the independent random numbers located at $0\sim 1$, K is the max iteration. The framework of the PSO-MLP is shown in Fig. 3.

Datasets

The DEAP (Dataset for Emotion Analysis using Physiological signals) dataset (https://paperswithcode.com/dataset/deap, DOI: 10.1109/T-AFFC.2011.15) was utilized to examine the emotional states of students based on their physiological responses while watching music videos. The dataset includes multimodal physiological signals from 32 participants, including EEG, GSR, heart rate, and facial expression data. Initially, the dataset was meticulously organized to ensure that the type of physiological signal systematically categorized each participant’s data. This preparation phase was crucial for maintaining consistency throughout the analysis.

In the preprocessing stage, various techniques were applied to clean and normalize the data. For EEG data, a bandpass filter was used to remove noise and artifacts, followed by normalization to achieve zero mean and unit variance. The EEG signals were then segmented into non-overlapping windows, typically one second long, to facilitate further analysis. Similarly, GSR, heart rate, and facial expression data were cleaned by removing outliers and artifacts, with segmentation performed in sync with the EEG data to maintain temporal alignment. We used the publicly available DEAP dataset, which contains EEG recordings from 32 participants across 32 video stimuli. To preprocess the data, the following steps were applied:

Filtering: EEG signals were bandpass filtered using a 4th-order Butterworth filter in the frequency range of 4–45 Hz to remove low-frequency drift and high-frequency noise.

Normalization: Each EEG channel was standardized using z-score normalization, computed across the entire signal of each channel per trial.

Segmentation: The continuous EEG signal was divided into overlapping segments using a 4-s window with 50% overlap, resulting in windows of 512 samples (at a sampling rate of 128 Hz).

Feature extraction: From each segment, we extracted statistical and frequency-domain features, including mean, standard deviation, skewness, kurtosis, and band power (delta, theta, alpha, beta, gamma bands).

Following preprocessing, feature extraction was performed to quantify and grade the physiological signals. For EEG data, features such as power spectral densities, band powers, and Hjorth parameters were extracted. Time-domain features, including mean, variance, standard deviation, skewness, and kurtosis, were derived from GSR and heart rate data. Sliding window techniques, with a window size of 5–10 s and a 50% overlap, were implemented to capture temporal changes in continuous signals, such as GSR and heart rate. These extracted features were statistically analyzed within each window to provide a comprehensive overview of the participants’ physiological responses during the video sessions.

The next phase involved classifying and labeling the data based on the extracted features. The emotional states of students were categorized into three distinct groups: Active, Negative, and Exhausted. This classification was achieved using supervised machine learning algorithms trained on labeled data. The categorization allowed for a detailed analysis of each participant’s class status, specifically identifying their levels of enthusiasm and fatigue. By mapping the physiological features to the defined class labels, the study was able to discern patterns in student engagement and emotional responses.

Finally, the processed data were analyzed to conclude the relevance to educational reform. The statistical analysis focused on understanding the correlation between physiological signals and class engagement levels, identifying patterns that indicated high enthusiasm, moderate participation, or signs of fatigue. These insights were then utilized to develop recommendations for optimizing class structures, teaching methods, or scheduling breaks to enhance student engagement and reduce fatigue. Cross-validation techniques were employed to assess the robustness of the classification model, with performance metrics such as accuracy, precision, recall, and F1-score being calculated.

Experimental setup

All experiments were conducted using a Python-based implementation of the PSO-MLP model. The computational environment was configured on a workstation equipped with an Intel Core i7-12700K CPU (3.6 GHz, 12 cores), 32 GB of RAM, and an NVIDIA GeForce RTX 3080 GPU with 10 GB of VRAM, running Windows 11 Pro. Model training and evaluation were performed using Python 3.9, with key libraries including PyTorch 1.13, NumPy 1.23, sci-kit-learn 1.1, and SciPy 1.10. The optimization process, utilizing PSO, was implemented with custom modules and validated against benchmark results. All code was executed within the Anaconda environment, and experiments were repeated five times to ensure statistical stability.

Table 1 outlines the architecture hyperparameter values for MLP and the optimization results using PSO. For the experiments, the parameters were set as follows: MLP had an input neuron count of 14, two hidden layers, and a rectified linear unit (ReLU) activation function for the hidden layers.

In this experimental training, due to the high number of parameters and limited training samples, overfitting is prone to occur. Therefore, a Dropout layer is applied to the input to mitigate this issue. During training, each input neuron is retained with a probability P, while the others are temporarily deactivated. This approach ensures that the trained model does not overly rely on specific local features, thereby enhancing its generalization performance.

The selection of 14 input neurons is based on the dataset’s input feature dimensionality, ensuring the network can adequately represent the input space. The choice of two hidden layers is motivated by the need for a balance between model complexity and training efficiency, with ReLU activation functions selected for their efficiency in mitigating vanishing gradient problems and facilitating faster convergence.

The Dropout_rate of 0.14 was determined empirically to strike a balance between preventing overfitting and maintaining model performance. A dropout rate that is too high could hinder the model’s ability to learn effectively, while a rate that is too low might not sufficiently address overfitting.

The batch size of 95 was chosen to fully leverage GPU capabilities and improve training speed while maintaining the stability of gradient updates. The value of 95 was found to provide a good trade-off between computational efficiency and convergence stability during preliminary experiments.

In the construction of the PSO-MLP model, the particle swarm algorithm optimizes parameters such as the number of neurons in the hidden layers, Dropout Rate, and Batch Size. The inertia weight ω = 0.5, and the acceleration constants c1 = c2 = 1 are also set for PSO.

We evaluated model performance using accuracy, precision, recall, and F1-score. These metrics were calculated using the following formulas:

(12) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\mathrm{T}\mathrm{P}/(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P})$$

(13) $$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\mathrm{T}\mathrm{P}/(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N})$$

(14) $$\mathrm{F}1\text{-}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\times (\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})/(\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}).$$

We applied five-fold cross-validation to ensure robustness. All metrics were averaged across the five folds. The same validation strategy was applied uniformly to both the baseline and proposed models for a fair comparison.

Learning status recognition with different modal data

The identification was carried out based on the collected data the results are provided in Fig. 4.

According to the identification results using different modal data in Fig. 5, it can be observed that when single modal data is used for identification, the differences between the results of the three types of states are small, and the results of the two states in different states are not significant. However, when the survey information and health information are combined and input into the model simultaneously, the recognition rate improves to a certain extent. It can be seen that although achieving good results in a single mode is difficult, the advantages of MLP become apparent when information is combined.

To provide a more detailed explanation of the recognition process and to inform the subsequent education reform, we also analyzed the recognition rate at different times of day during the experiment. For the morning, afternoon and night, it is consistent with the overall results. The combined features have achieved a high recognition rate, with accuracy exceeding 90%. Overall, the recognition rate for the evening session is slightly lower than that of the other two time periods. This is because the students in the evening session, due to the matching of the day, had a large deviation when filling in the survey. This error was identified through discussions with the students themselves, suggesting that we can redistribute the weight of data from different modes when entering future research to ensure a high recognition rate. On this basis, to make a detailed analysis of the current education, we also analyzed the proportion of students in different learning states in three periods of the day, and the results are shown in Fig. 6.

The proportion of students in different periods, as shown in Fig. 6, is generally consistent with expectations: in the morning, students are more active, while those who are tired and negative account for a relatively small proportion. With the continuous increase in learning time per day, the proportion of students who are active decreases. However, it is worth noting that the proportion of students who are exhausted is the least at night, which deviates slightly from expectation. After communicating with students, it was found that most evening courses are elective or relatively low-difficulty courses. After a long rest in the evening, students’ enthusiasm has recovered to a certain extent, resulting in the lowest proportion of exhausted courses. However, the proportion of active courses has increased compared to the noon courses.

Figure 7 presents the confusion matrix for the emotion recognition task. The results indicate that the method accurately identifies high arousal and moderate valence emotions, with accuracies of 90% and 94%, respectively. However, the accuracy for identifying low arousal and low valence emotions is lower, at only 61% and 58%, respectively. This discrepancy is due to the significant physiological changes typically associated with high arousal and high valence emotions, such as surprise and fear during stimulation, which leads to higher recognition accuracy. In contrast, the physiological changes associated with low arousal and low valence emotions are less pronounced, resulting in lower recognition accuracy.

Comparison of the PSO-MLP and other methods

To verify the effectiveness of the proposed approach, we compared it with several existing methods, as shown in Fig. 8.

According to the comparison of different methods in Fig. 8, when the PSO method is not used for optimization, MLP does not have an advantage in the overall recognition rate. Its accuracy is lower than that of SVM and higher than that of the traditional NN method. The observed 0.029 accuracy gap between PSO combined with an MLP and SVM can be attributed to several underlying factors. Firstly, the inherent characteristics of the algorithms play a crucial role in their performance. PSO is a metaheuristic optimization technique inspired by swarm behavior and is used to optimize the weights and biases in MLPs. While MLPs can model complex patterns through their layered architecture, their performance is highly dependent on the tuning of hyperparameters and network structure.

In contrast, support vector machines (SVMs) are designed to find an optimal hyperplane that separates different classes and are generally effective in high-dimensional spaces. However, their performance can be sensitive to the choice of kernel function and its parameters. Table 2 summarizes the classification performance of the proposed PSO-MLP model compared to baseline models (MLP and SVM), including average metrics, standard deviations, 95% confidence intervals, and statistical significance (p-values).

However, with parameter optimization using PSO, the overall recognition rate has been further improved, as PSO helps the MLP avoid falling into local extreme values, thereby increasing the recognition accuracy even further.

Conclusions

The integration of artificial intelligence (AI) in educational settings, particularly in mobile and online learning environments, has ushered in transformative opportunities. This study introduces an intelligent method for assessing students’ learning states using multimodal physiological data, aiming to enhance the effectiveness and efficiency of remote education. By leveraging the joint characteristics of pre-class mental surveys and health big data from teachers and students, our proposed PSO-MLP model offers a robust solution for accurately classifying students’ enthusiasm levels during classes. Through experiments utilizing the DEAP dataset and employing a PSO-optimized MLP neural network, we achieved a significant recognition accuracy of 89.1%, surpassing traditional methods. This approach not only identifies students’ states (Active, Negative, Exhausted) based on physiological signals, such as EEG, GSR, and facial expressions, but also demonstrates the model’s effectiveness across different times of the day, with peak recognition rates exceeding 90%. Moreover, the study compares our PSO-MLP model against other methodologies, showcasing superior performance in recognition rates due to PSO’s optimization capabilities, which mitigate local minima issues inherent in MLP training. This optimization ensures the robustness and applicability of our model in real-world educational contexts, supporting personalized teaching approaches and effective student health management.

Furthermore, our findings contribute essential insights into the challenges of remote education, particularly in gauging student engagement and well-being. By integrating advanced AI tools, such as MLP, enhanced by PSO, educators can make informed decisions to foster more interactive and supportive learning environments. This technological integration not only monitors but also enhances the overall well-being of both educators and students, thereby catalyzing educational reform efforts.

Limitations

While the proposed method demonstrates promising performance, several important limitations must be acknowledged to present a more balanced assessment. First, the model’s effectiveness relies heavily on the quality, representativeness, and diversity of the physiological data used. Although the DEAP dataset provides a solid foundation, it is collected under controlled experimental conditions. It may not fully capture the complexity, variability, and contextual diversity of real-world educational environments. This constraint limits the generalizability of the findings, especially when considering diverse student populations, varying emotional baselines, and environmental factors. Second, although the PSO-MLP model achieves high accuracy, its complexity introduces challenges related to interpretability. Neural networks, particularly those optimized through metaheuristic algorithms like PSO, often behave as “black boxes,” making it difficult for educators or researchers to understand how predictions are made. Without clearer insights into feature importance or decision pathways, the practical application of the model in education may be hindered, as stakeholders may lack the necessary transparency to trust or act on the model’s outputs.

Furthermore, the risk of overfitting must be considered. While cross-validation helps mitigate this, relying on a single dataset raises the possibility that the model may have adapted too closely to dataset-specific patterns rather than learning generalizable features. Additionally, there is a limited exploration of how performance varies across demographic subgroups or recording sessions, which could introduce unintended biases or instability. Finally, the deployment of such a system in real-time educational settings presents practical challenges. Real-time processing of multimodal physiological signals requires substantial computational resources, and system latency can compromise responsiveness and usability in dynamic classroom environments. These limitations underscore the importance of future work involving external validation, interpretability enhancement, and domain adaptation to bridge the gap between experimental performance and real-world utility.