Application of multimodality perception scene construction based on Internet of Things (IoT) technology in art teaching

Image feature extraction

Facial expression recognition can analyze students’ learning dynamics and teachers’ teaching status, judge classroom emotions, and objectively evaluate classroom effects. Facial expression recognition technology for classroom interaction and emotional evaluation has become a hot issue in intelligent education, which has significant exploration and application value. This study constructs a multimodal perceptual scene, using facial expression recognition as one of the interaction methods, which can recognize the expressions of teachers and students and analyze and evaluate the classroom teaching situation according to the recognition data to adjust the teaching strategy.

This research uses the DenseNet-BC convolution neural network to extract image features. DenseNet network adopts a dense connection mode. It does not need to re-learn redundant feature mapping. It has the advantages of reducing gradient disappearance, strengthening feature transmission, and using features efficiently. DenseNet-BC network includes the Bottleneck layer and the Transition layer. The transition layer is a network layer composed of roll-up and pooling layers. DenseNet-BC network includes three Dense Blocks and two transition layers. Figure 1 shows the structure of the DenseNet-BC network, where Dense block n represents the nth dense block.

This study selected a variety of classifiers for experiments, including classical classifiers such as support vector machine (SVM) and random forest (RF). At the same time, considering that when video samples are divided into image samples, extended sequence image frames are obtained, and long-short-term memory (LSTM) has significant advantages in processing long sequence data, a classifier based on LSTM is designed for the experiment. The structure of the classifier based on LSTM is shown in Fig. 2. The input sequence X is a feature of different times. A batch standardization layer is added after the input layer. The LSTM array composed of multiple LSTM nodes captures the feature information. The feature information of different times is averaged through the average pooling layer and output to the Softmax layer for classification.

In multimodality emotion recognition, weighted voting methods and weighted average methods are common decision fusion methods. Among them, the voting method is more suitable for the case where the models in decision fusion are independent. Considering that the training of each model in this study is independent and there is no strong dependence, using the weighted voting method for decision fusion may bring some improvement. The weighted voting method is implemented explicitly as follows:

Suppose the number of expression categories is M, the number of models is L, h_i is the i^th emotion recognition model, and w_i is the contribution weight of the ith model to the decision results of the fusion model, where the constraints of w_i are: (1)${\displaystyle \left\{\begin{array}{c}\sum _{i=1}^L{w}_i=1\hfill \\ w_i\in \left[0,1\right],i=1,2,\dots ,L\hfill \end{array}.\right.}$

For sample x, let f(x) be the set of weighted voting values of various expression categories based on the weighted voting method, and y(x) be the decision result of expression categories, then: (2)${\displaystyle \begin{array}{c}\hfill f_j\left(x\right)=\sum _i^L{w}_{i}I\left(h_i\left(x\right),j\right)\hfill \\ \hfill y\left(x\right)=argmax{f}_j\left(x\right)\hfill \end{array}.}$

where j = 1, 2, …, M, the definition of indicating function I is: (3)${\displaystyle I\left(a,b\right)=\left\{\begin{array}{c}1,a=b\hfill \\ 0,a\ne b\hfill \end{array}.\right.}$

This research is mainly based on the AFEW data set for multimodality emotion recognition research. Firstly, the video data of the AFEW dataset is divided into audio and picture data, and emotional features of voice and image modes are extracted, respectively. When extracting image features, we must first perform two preprocessing operations: face detection, correction, and clipping. Since the author of the AFEW dataset has provided most of the cropped grayscale facial images, the image data not provided are only 17 videos in the Train training set and 12 videos in the Val verification set. Therefore, we only preprocess the image data that is not provided. After successfully extracting the face gray image, we still need to carry out histogram equalization to reduce the impact of light on the image. After the preprocessing operation is completed, we fine-tune the pre-trained DenseNet-BC convolutional neural network model on the FRE2013 dataset and then extract image features using the preprocessed image as the input of the fine-tuned model. When using the DenseNet-BC convolution neural network to extract features, take the output of the last average pooling layer of the DenseNet-BC network as the feature, which is marked as DenseNet-pooling3.

Speech feature extraction

SoundNet is a deep convolutional neural network with a high learning ability of voice information (Aytar, Vondrick & Torralba, 2016). The structure and implementation schematic diagram of the SoundNet network are shown in Fig. 3.

First, video is cut into audio and RGB image frames. RGB image frames are recognized and classified by using the image-like convolutional neural network (ImageNetCNN) and scene-like neural network (Places CNN), respectively, and the results of RGB image frame classification are used as the supervision information of SoundNet network, from which the voice-related information can be learned. The SoundNet network used in this study consists of eight layers of convolution layer and three layers of pooling layer, and the loss function is KL divergence.

Before extracting speech features, the SoundNet network preprocesses audio files through three essential operations: resampling, framing, and windowing. Resampling standardizes the sampling rate across all audio inputs, ensuring uniformity for subsequent processing steps. Framing divides the audio signal into short, overlapping segments to capture temporal dynamics effectively, preserving transient sounds and temporal information. Finally, windowing applies a window function to each frame, such as the Hamming window, to minimize spectral leakage and enhance the accuracy of spectral analysis by smoothing frame edges and reducing artifacts. Together, these preprocessing steps ensure the audio data is appropriately structured and represented for extracting meaningful speech features, enhancing the network’s ability to understand and process audio inputs effectively.

When using the SoundNet convolution neural network to extract features, the original data of the audio file is taken as the input, and the extracted features are marked as SoundNet. The SoundNet network structure and implementation schematic diagram used in this study are shown in the figure, where Conv n represents the nth layer of convolution, and pool n represents the nth layer of pooling.

Configuring the Mel filter bank involves determining the parameters of each filter in the bank based on the Mel scale. The Mel filter bank aims to mimic the human auditory system’s response to different frequencies. Each filter in the bank is defined by its center frequency, which is spaced logarithmically along the Mel scale. The extraction process of Mel frequency cepstrum coefficient MFCC is as follows: First, FFT is performed on a sampled frame of discrete speech sequence x (n). The formula of FFT is as follows: (4)${\displaystyle X\left(k\right)=\sum _{n=0}^{N-1}x\left(n\right)e^{-j\frac{2\pi }{N}nk},k=0,1,2,\dots ,N-1}$

where N is the frame length.

Configure the Mel filter bank and calculate the filtered output. The frequency response Hm(k) of the Mel filter is: (5)${\displaystyle H_m\left(k\right)=\left\{\begin{array}{c}0,k<f\left(m-1\right)\hfill \\ \frac{2\left(k-f\left(m-1\right)\right)}{\left(f\left(m+1\right)-f\left(m-1\right)\right)\ast \left(f\left(m\right)-f\left(m-1\right)\right)},f\left(m-1\right)\le k\le f\left(,\right)\hfill \\ \frac{2\left(f\left(m+1\right)-k\right)}{\left(f\left(m+1\right)-f\left(m-1\right)\right)\ast \left(f\left(m\right)-f\left(m-1\right)\right)},f\left(m\right)\le k\le f\left(m+1\right)\hfill \\ 0,k\ge f\left(m+1\right)\hfill \end{array}\right.}$

where f(m) is the center frequency of the filter.

Then, calculate the logarithmic energy S(m) output by each filter bank. (6)${\displaystyle S\left(m\right)=ln\left(\sum _{k=0}^{N-1}{\left|X_a\left(k\right)\right|}^2{H}_m\left(k\right)\right),0\le m\le M\left(6\right)}$

where, M is the number of filters. $X_a\left(k\right)$represents the input signal in the frequency domain.

Finally, MFCC coefficient C(n) can be obtained by DCT of discrete cosine transform, and the formula is described as follows: (7)${\displaystyle C\left(n\right)=\sum _{m=0}^{N-1}S\left(m\right)cos\left(\frac{\pi n\left(m-0.5\right)}{M}\right),n=1,2,\dots ,L}$

where L is the MFCC coefficient’s order, cos denotes the cosine function. This process typically results in a set of MFCC coefficients representing the features of the input signal that are relevant for speech or audio processing tasks.

Sensor technology

This research uses temperature, humidity, somatosensory control, vibration, and other types of sensors to build a multimodal perception scene. Each sensor corresponds to different senses of people to obtain various information and provide interaction interfaces between users and systems.

Through digital temperature and humidity sensors, analog signals are measured and directly converted into digital signals, which are used to measure and control air temperature and humidity information in the multi-mode perception scenario. The Kinect body sensor is used to capture gesture information from users. Kinect uses a camera to scan the real environment as a whole. In the digital art flower cultivation system, the body sensor captures the user’s flower-picking action. Then, the system simulates the scene where the user touches the flower and makes the petals fall. The odor generator can feed the corresponding odor to the user according to different scenes, stimulate the user’s sense of smell, and deepen their perception of the screen. In the digital flower culture system, the somatosensory control sensor recognizes the user’s hand movements and then triggers the aroma generator to work, releasing the fragrance of flowers. Vibration and sound feedback sensors provide users with tactile and auditory stimuli, including playing prompt sound effects and outputting vibration. In the digital flower cultivation system, the sound feedback sensor outputs the preset process guidance sound effect during the user’s participation in the course tasks, such as “Welcome to the art teaching class. Do you want to cultivate a rose?”, “Don’t worry, water the soil first; the seeds will sprout!”, and so on. It can also output natural sound effects. For example, when the flowers are in full bloom, they can output the sound effects of bees flying to increase the information feedback on the user’s hearing so that users can immerse themselves in it. The pyroelectric infrared sensor is used to detect the existence of human beings by detecting infrared rays. In the digital art flower cultivation system, the system will automatically identify the user’s position and give feedback on things such as breeze, fragrance, and vibration, according to the scene settings. Each sensor module is shown in Fig. 4.

Construction of multimodality perceptual scene system

The realization of multisensory interaction is to achieve reasonable participation in interactive behavior through reasonable design of interactive tasks and finally form a good interactive experience. The design of interactive tasks is the first problem to be solved in the multisensor interaction model. Interactive task design needs to arrange task flow and specific ways of task participation according to the impact on integrating participants’ sensory channels. The system controls the visual channel so that the channel integration process presents the effect of visual information leading and other sensory assistance participating. In addition, the system needs to design the auditory channel information, including system prompt tone, music, sound, etc., and the time and location of the sound generation device.

This research builds a multimodality perception situational teaching system based on IoT technology. Students can immerse themselves in the art classroom through visual, auditory, tactile, olfactory and other interactive channels. Following the organization mode of human–computer interaction, the sensors are arranged in a reasonable space. The sensors correspond to all human senses, making the interaction process simple and natural. This research takes the interactive prototype system of cultivating digital art flowers as an example. The digital art flower cultivation system includes water level sensors for watering amounts, temperature and humidity sensors for acquiring temperature and humidity signals, and wind sensors for acquiring wind direction signals. The temperature sensor receives temperature data and controls seed germination; the humidity sensor obtains the humidity data and controls the bud to blossom; the wind sensor receives wind data and controls the movement direction of digital flowers.

The display device outputting visual information should be placed in the front, the odor-generating device outputting odor should be placed in the front of the user and lowered, the audio device outputting sound on both sides of the display model screen should be set and the somatosensory control device outputting gesture and other operations should be placed in the front and close to the user; each sensor of input data should be distributed at the place where people can reach their left and right hands. This spatial layout allows users to participate in multisensory interaction naturally and comfortably and obtain a more natural interaction experience than single-sensory interaction.

According to the relationship between the task and the system, a system state transition diagram can be obtained, as shown in Fig. 5. The yellow diamond represents the decision maker of information fusion. In the input phase, the decision maker completes the accurate judgment of the input sensory information, triggers the system to switch states, and makes decisions on the output content and output time weight of each state according to the input information to accurately transmit the information to the user’s senses, The condition of sensory channel integration is formed, which leads to channel integration and completes the interaction process.

The interactive task of students’ participation in training digital art flowers is shown in Table 1.

The interactive task of cultivating digital art flowers designed in this study includes two paths. The specific steps of path one include: through watering action, students trigger the water level sensor to make the seeds germinate; through the action of covering heat, students trigger the temperature sensor to make the buds bloom; through the blowing action, students can trigger the wind direction sensor to make the flowers swing; by touching, students can trigger the body sensor to make the petals fall. The specific steps of Path 2 include sprouting seeds through voice interaction, making buds blossom through recognizing smiling expressions, making flowers swing through gesture recognition, and dropping petals through sad expressions. When the seeds germinate, the students will hear the sound of watering. When the buds bloom, the sensor signal is transmitted to drive the motor to rotate. At the same time, the spice box is opened, and the motor blows air into the spice box to diffuse the gas molecules. The smell is transmitted to the people. At this time, you can see the flowers blooming, smell the flowers, hear the bees flying, and see the petals falling one by one when the petals fall.

Evaluation of image and speech recognition models

After model training and testing, the classification results of facial expression recognition based on this study’s DenseNet-BC convolution neural network are shown in Table 2.

The analysis of experimental results reveals a noteworthy improvement in accuracy, approximately 5% when comparing the LSTM-based classifier with the support vector machine (SVM) classifier utilizing image features—DenseNet pooling 3, VGG conv13, and VGG fc1. Notably, the highest accuracy in expression classification is observed with the DenseNet pooling features. Consequently, this study combines DenseNet pooling3 features and the LSTM classifier to underpin the multimodal perceptual scene construction algorithm.

Expanding on speech features involving MFCC and SoundNet, both SVM and LSTM-based classifiers are employed for speech recognition purposes. The classification outcomes are presented in Table 3.

The proposed LSTM+SVM model presents distinct advantages in art education, notably surpassing traditional classifiers with a notable 6% accuracy improvement, particularly evident when analyzing speech features like MFCC and SoundNet. By incorporating MFCC speech features with the LSTM classifier, the model achieves a remarkable accuracy rate of 96.15%, showcasing its ability to integrate multimodal information for enhanced scene construction effectively. This combination demonstrates the LSTM’s prowess in capturing temporal dependencies and nuanced patterns within data. It underscores its suitability for tasks requiring complex scene understanding, which is essential in art education contexts. Moreover, the LSTM’s capacity to handle temporal dynamics and adapt to varying input sizes and resolutions further solidifies its position as a superior choice for applications necessitating nuanced analysis and comprehension of artistic expressions.

Usability evaluation of multimodality perceptual interaction system

Usability evaluation experiments are devised to assess various indicators across each stage of users’ engagement with the multimodal awareness situational system. These indicators encompass user operation time, frequency of actions, occurrence of errors, system response time, and instances of system exceptions. The objective is to evaluate the system’s responsiveness, feedback mechanisms, fault tolerance, ease of use, and facilitation of learning, among other aspects.

Responsiveness and feedback can be quantified as response time and feedback data; fault tolerance can be quantified as the number of system exceptions and user errors; low difficulty can be quantified as the number of movements; easy to learn can be quantified as the time interval between system prompts and user operations to verify the availability of multimodality perceptual situational systems.

• User learning time t₁: Start time of user watering operation—end time of prompt tone;

• System response time t₂: Prompt tone start time—start time of user watering operation;

• User learning time t₃: Start time of user heating operation—end time of prompt tone;

• System response time t₄: Prompt tone start time—start time of user’s hot operation;

• User learning time t₅: Start time of user blowing operation—end time of prompt tone;

• System response time t₆: Observe whether the system reacts immediately after blowing to indicate the responsiveness of the system to the task;

• Number of system exceptions: Number of system non-responses+number of system response errors.

Based on the data of 20 experimenters, this study used SPSS data analysis software to analyze the original data results, as shown in Table 4. Finally, it calculated the average learning time of users, the average response time of the system and the corresponding standard deviation, as shown in Table 5.

By analyzing the mean and standard deviation, we can determine the system’s low difficulty, easy learning habits, and fault tolerance. Based on the analysis of the above data results, we can draw the following conclusions: first, in terms of the average learning time of users, the system relies on its voice prompts to inform users of the operation method. The user’s learning time is within 2s, and the standard deviation is minimal. The data fluctuation is very small, which indicates that a large number of users’ learning time is close to the average value, and the user’s learning cost is meager, so it can be seen that the system is easy to learn. In terms of the average response time of the system, the average response time of the task is 6.453s, and the standard deviation is 0.129, indicating that the system response time on this task is stable at about 6.45s, with slight fluctuation, which ensures the stability of the system response time and the continuity of user operations. The user’s subjective evaluation results also reflect that the user is delighted with selecting this time threshold. By observing whether the user’s blowing behavior immediately caused the system response of flower swaying, the results showed that 20 testers performed well in the task.

Table 6 shows the average user operation errors and system exceptions. The average number of users’ operations is 3.1, which reflects the low difficulty of the system, while the average number of system exceptions is only 0.09, which reflects the strong fault tolerance of the system.

In this study, the subjective evaluation scale evaluates the realism, interactivity and content richness of the multimodality perception scenario system. The scoring results of each indicator are shown in Table 7.

From the statistical results of the subjective evaluation scale, the multimodal perception scenario system strengthens the expression of information, enriches the level of information, makes up for the lack of single sensory information, improves the user’s sense of pleasure, and achieves a more realistic and natural user experience. Adding olfactory channel feedback enhances flowers’ authenticity and the system’s interactivity. This shows that the multisensor channel stimulation combined with the visual and auditory sense and olfactory sense of the system brings users a better sensory experience. The smell generation function has significantly improved the user experience of the whole system. Overall, it can be concluded that the system’s information authenticity, interactivity and content richness are good.