Design of a 3D emotion mapping model for visual feature analysis using improved Gaussian mixture models

Keyframe extraction based on machine vision

The process begins with extracting keyframes from multimedia content to identify representative frames. This is achieved by performing optical flow calculations, where each frame’s entropy value is computed. Frames with the highest entropy values are selected, ensuring that the most informative content is retained for further analysis. Keyframe extraction is selecting the most representative and informative frames in a multimedia sequence to reduce storage and processing costs while preserving important multimedia content information. Firstly, we perform optical flow calculations on the relevant multimedia content sequences, represented as follows:

(1) $$E(w)=\beta E_{color}(w)+\gamma E_{grad}(w)+\alpha E_{smoorh}(w)$$where $E_{color}(w)$ represents the assumption of brightness invariance; $\alpha$, $\beta$ and $\gamma$ represent adjustable weight parameters, respectively. By introducing gradient constraint $E_{grad}(w)$ to reduce the impact of lighting, and using $E_{smoorh}(w)$ to smooth the video. Calculate the entropy value corresponding to the current optical flow chart using Formula (2).

(2) $$E_{-}img=-\sum _k^m{\log }_2{p}_{k}E(w)$$where $E_{-}img$ represents entropy value; $m$ represents grayscale level; $p_k$ represents the proportion of pixels with a grayscale value of k in the image. The larger the entropy value, the more information in the image.

As the primary step for extracting multimedia energy features in multimedia signal processing, we initiate the process by windowing and framing the multimedia signal $x(j)$, obtaining the k^th frame multimedia. The multimedia signal is stored in an array y of length N, with each window sampled at a length of wlen and a sampling frequency of fs. Olap=wlen-dis describes the overlapping part between two frames, while dis represents the displacement between consecutive frames. This framing process is based on Eq. (3) and is applied to the multimedia signal of length N. It lays the foundation for subsequent feature extraction and the mapping of emotional space in our design based on visual aesthetic features:

(3) $$fs={\displaystyle \frac{N-olap}{dis}={\displaystyle \frac{N-wlen}{dis+1}.}}$$

Then, by applying Eq. (4), the average amplitude of the video is calculated to obtain the energy feature corresponding to the video:

(4) $$\{\begin{array}{c}{y}_k(j)=win(j)\times x[dis(k-1)+j]E\mathrm{\_}img\hfill \\ M(k)=\sum _{j=0}^{L-1}|y_k(j)|fs\hfill \end{array}$$where $y_k(j)$ represents the values of a single frame, $win(j)$ denotes the window function, and $M(k)$ signifies the energy magnitude corresponding to a video frame.

The process mentioned above achieves the alignment of entropy sequences and feature sequences. Subsequently, a multiplication operation is performed between the entropy sequence and the visual feature sequence to accomplish feature fusion, acquiring entropy sequences associated with the visual aspect.

Calculate the entropy sequence corresponding to the fused features. Using the following formula, frames with values exceeding the threshold are selected as key frames for the visual aspect by comparing the obtained values with a threshold. The expression for extracting key frames is as follows:

(5) $$V={\displaystyle \frac{|H_{current}-H_{key}|M(k)}{H_{ky}}}$$where $H_{key}$ and $H_{current}$ are the frame values corresponding to key frames and the entropy values corresponding to the current frame, respectively.

Feature extraction

Working sequence potential function

Once the keyframes are extracted, we establish a working sequence potential function using the Harris3D operator. This function identifies and evaluates motion-related visual features in the video, facilitating the recognition of specific actions or emotions. The function calculates the shortest Euclidean distance between local reference points and spatiotemporal points in the video, allowing the system to detect relevant motion sequences efficiently.

Utilizing the Harris3D operator based on motion recognition results to establish the sequence potential function, laying the foundation for visual feature extraction.

Let $(X_{zi},Y_{zi})$ represent the key points in the video, with the local reference point as $(a_i,b_i)$ and n denotes the shortest Euclidean distance between the local reference center point and the spatiotemporal point, and its calculation formula is as follows:

(11) $$n={\displaystyle \frac{\arg min\sqrt{{(a_i-x_j)}^2+(b_i-y_j)}}{(x_{zi},y_{zi})}}$$where $(x_i,y_i)$ represents the spatiotemporal point.

Obtain the action dataset through k-means clustering, where $f_p$ represents the BOW feature corresponding to the action image p. The calculation method is:

(12) $$f_p={\displaystyle \frac{K_n\times N}{K}\times {\displaystyle \frac{K_n\times 162}{p}}}$$where N represents the spatiotemporal unit sequence length corresponding to the action image, and Kn represents the number of cluster centers within the n range.

Fuse the BOW features through the following expression:

(13) $$F_p=\sum _{n\in [1,7]}{K}_n\times N{f}_p$$where $F_p$ represents the fused features within various levels of the action image.

Now, based on the above analysis, we can obtain the working conditional probability model as follows:

(14) $$P(Y,h/X,\theta )=\{X_i{\}}_{i=1}^t{F}_p\times {\displaystyle \frac{\exp (f_p\cdot \phi (Y,h,Z))}{\sum (F_p\cdot \phi (Y,h,X))}}$$where $\phi (Y,h,X)$, $Y$, $\theta$, $h$, and $\{X_i{\}}_{i=1}^t$ represent the sequence potential function of actions, the sequence label, represents a constant, represents the hierarchy, and any action sequence, respectively.

For different levels of the working sequence potential function, the calculation is performed as follows:

(15) $$\phi (Y,h,X)=[\sum {\phi }_1(X_j,h_j)\sum {\phi }_2(Y,h_j)]{\displaystyle \frac{\sum {\phi }_3(Y,h_j,h_k)}{P(Y,h/X,\theta )}}$$where ${\phi }_1(X_j,h_j)$ represents the relationship between the prediction node and the latent variable node and ${\phi }_2(Y,h_j)$ represents the relationship between the sequence label point and the latent variable node.

Let $(x_1,y_1)$, $(x_2,y_2)$, and $(x_N,y_N)$ represent the sample dataset, where $x_i$ denotes the sample data, and $y_i$ represents the sample labels. Utilizing the AdaBoost algorithm based on calculating the sample error rate.

(16) $${\epsilon }_t={\displaystyle \frac{\phi (Y,h,X)\times (h_t(x_i)\ne y_i)}{(x_1,y_1),\cdots ,(x_i,y_i),\cdots ,(x_N,y_N)}.}$$

Based on the above results, extract visual features to obtain:

(17) $${\hat{C}}_i=\arg min||d_i-C(d_j{)}^2||\times {\epsilon }_t$$where $d_i$ represents the j-th feature data present in the aerobics jumping action sequence, $C(d_j)$ represents the category to which the j-th aerobics jumping feature data belongs in the training sample. Utilizing the model constructed by the above formula, complete the extraction of visual features.

Three-dimensional color-emotion space model

Before establishing the model, it is essential to smooth and normalize the feature data to ensure the model’s robustness. This process involves normalizing the data using the maximum-minimum scaling method.

(18) $$r^{\prime }={\displaystyle \frac{r-min(X)}{max(X)-min(X)}}$$where the least and maximum ratings in dataset X are denoted, respectively, by min(X) and max(X).

Building models for visual texture aesthetics represents a distinctive application of machine learning. These models aim to bridge the gap between high-level aesthetic perceptions and low-level texture properties. In this section, we utilize six activation functions to construct the emotional model, comprising the emotion, judgment, and perception layers.

The primary function of the emotion layer is to extract emotional information from the input visual aesthetic features. Within the three-dimensional color-emotion space mapping model, the emotional layer perception model is an intermediary between high-level emotional perception and low-level texture features. Its construction is detailed as follows:

(19) $$G=F_1\left(M\right)+R_0$$

The primary function of the judgment layer is to make assessments or evaluations, thereby identifying or quantifying the position or attributes of input visual aesthetic features in the emotional space. The judgment layer perception model plays a role in categorizing, scoring, or positioning visual features, contributing to the understanding and expressing emotional information associated with these features. Its construction involves:

(20) $$T=F_2\left(M\right)+F_3\left(G\right)+R_1$$

The primary function of the perception layer is to capture and represent emotional information related to visual aesthetic features in the emotional space. It focuses on extracting emotional features from the input and understanding the distribution and relationships of these features in the emotional space through model construction. The specific construction involves:

(21) $$Q=F_4\left(M\right)+F_5\left(G\right)+F_6\left(T\right)+R_2$$where $F_i(i=1,2,3,4,5,6)$ represent six linear/non-linear activation functions, $R_0$, $R_1$, and $R_2$ represent emotional thresholds, i.e., the minimum values of aesthetic attribute scores; the symbol + indicates the accumulation of emotions through different perceptual stages.

This section employs a comparative method to calculate different sets of emotions and construct an emotional space model. After representing emotions in a three-dimensional space, we calculate the average angle of each emotion to its origin. This angle defines the shortest rotation angle for any emotion around the origin and is responsible for assuming the establishment of emotions.

To compute the emotional space in three dimensions, we calculate the average angle for each emotion to construct an emotion wheel and obtain definitive patterns. To maintain normality, we also exclude some outliers and data points unusually distant from other values in the dataset. The standard formula defines the angle between two vectors.

(22) $$\begin{array}{rl}\cos \mathrm{\Theta }(\mathrm{U},\mathrm{V})& ={\displaystyle \frac{DotProduct}{EucledianNorm}}\\ \cos \mathrm{\Theta }\left(\mathrm{U},\mathrm{V}\right)& ={\displaystyle \frac{{(U,V)}_R}{||\mathrm{U}||||\mathrm{V}||}}\\ \mathrm{\Theta }& =\mathrm{A}\mathrm{r}\mathrm{c}\mathrm{C}\mathrm{o}\mathrm{s}(\cos (U,V))\end{array}$$where $(U,V{)}_R$ defines scalar product for any pairs of vectors $U,V\in X_R$ in any real vector space $X_R$. The scalar product is defined as:

(23) $$(\mathrm{U},\mathrm{V}{)}_{\mathrm{R}}=\sum _{i=1}^n{U}_i{V}_i;\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}n\ge 2$$

And

(24) $$||U||=\sqrt{{(U,U)}_R};||V||=\sqrt{{(V,V)}_R}$$where |U|| and ||V|| denote the Euclidian norm or magnitude of respective vectors U and V.

Based on the above analysis, the flowchart of the proposed method in this article is shown in Fig. 1.

The critical steps in our proposed method are as follows:

1. Data collection and preprocessing:

A diverse dataset comprising various images annotated with emotional labels was collected. This dataset included images from publicly available sources and specifically curated data for our experiments.

2. Feature extraction using improved GMM:

We enhanced the GMM to handle high-dimensional visual data more effectively. This involved fitting a mixture of multiple Gaussian distributions to the feature space, capturing the underlying structure and complexity of the data.

3. Feature fusion using entropy sequences and visual feature sequences:

We integrated entropy sequences with visual feature sequences to enhance feature fusion. The entropy sequences capture the uncertainty and information content in the visual features, while the visual feature sequences provide detailed spatial and color information.

4. Construction of multi-layer emotion perception model:

Six activation functions were employed to construct a machine vision model with three primary layers: emotion layer, judgment layer, and perception layer. Each layer handles different aspects of emotion recognition, ensuring a holistic approach to modeling emotional states.

Experiment preparation

This study aims to design a three-dimensional color emotion space mapping model based on visual aesthetic features. Before experimenting, we undertook thorough preparations to ensure the reliability of our results. We selected a dataset comprising 200 multimedia contents, with 100 groups allocated to the training set and the remaining 100 groups designated as the testing set. We employed machine vision techniques to extract rich features from the multimedia content.

In the experiment, we defined the following metrics to evaluate performance:

Emotion mapping accuracy (EMA): Measures the precision of the model in mapping color features to the emotion space.

Emotion recognition precision (ERP): This represents the model’s accuracy in identifying specific emotion categories.

Emotion recognition F1 score (ERFS): Provides a comprehensive evaluation of the model’s overall performance in emotion recognition, considering precision and recall.

These metrics assess the model’s emotion mapping and recognition performance, offering a thorough and nuanced evaluation. By utilizing these metrics, we can better understand the model’s efficacy in mapping emotions within the context of visual aesthetic features.

Based on the selected seven visual features (color, composition, texture, clarity, depth, emotional content, and dynamic elements), denoted as $m_i(i=1,2,3,4,5,6,7)$, corresponding evaluation parameters are calculated.

(25) $$\begin{array}{rl}G(1)& =8\text{,}034.2356m_1+195m_2{m}_3+{\displaystyle \frac{18.23}{m_4}+34.56(m_5{m}_6+m_7)}\\ G(2)& =-2.4097+2.4427\times {10}^{-3}{m}_7\\ G(3)& =-1\text{,}249.34+1\text{,}034.2312m_2+201m_1{m}_4+{\displaystyle \frac{22.23}{m_3}+34.56m_7}\\ T(1)& =-8\text{,}012.2-2.36m_1+15.26m_7+G(3)+0.1285m_{3}G(1)\\ T(2)& =-120.36+14.22m_7+6.23m_2{m}_4-15.23m_3{\displaystyle \frac{5.214}{m_6}+15.23G(1)G(2)}\\ T(3)& =-203.56+10.563m_4+{m_5}^2+1.234G(1)m_3\\ Q& =38.4474-10.7260\cdot m_2-57.6083\cdot m_4+0.0914\cdot T(3)+1.4546\cdot m_1\cdot G(1).\end{array}$$

The corresponding evaluation parameters are demonstrated in Table 1.

Experimental results

Firstly, to validate machine vision methods’ effectiveness in emotional modeling visual aesthetic features, we employed three distinct methods for recognizing video action orientations. Method 1 is our proposed approach. Method 2 utilizes the Maximum Entropy Model combined with the local binary pattern (MEM-LB method): This method integrates maximum entropy principles with local binary patterns to create a robust framework for capturing visual features. Method 3 employs the Multi-Threshold Optimized method (MTO method): This method is designed to enhance the precision and efficiency of emotion recognition by optimizing the thresholds used in feature extraction and classification. The test results are presented in Fig. 2.

Analyzing the data presented in Fig. 2, it is apparent that when using the proposed method to monitor the working orientation of multimedia content, the resulting monitoring curve closely aligns with the actual angle curve. In contrast, when employing the MEM-LB method, the monitored angle tends to be higher than the exact angle. Similarly, the MTO method results in a monitored angle lower than the actual angle. This analysis demonstrates that Method 1 effectively identifies azimuth angles, as it extracts keyframes from multimedia content, enabling precise estimation of azimuth angles within these keyframes.

Secondly, we evaluated the learning convergence performance of the training set in the emotional modeling process across different methods (the proposed method, the convolutional neural network (CNN) method, and the long short-term memory (LSTM) method). The convergence processes over fewer iterations using these methods are depicted in Fig. 3.

Figure 3 illustrates the convergence speed plots under different methods. It can be observed that the machine vision method proposed in this article exhibits a faster convergence speed compared to the CNN and LSTN methods.

The parameter values during the iterative process are shown in Fig. 4.

Figure 4 shows that during the iterative process, the parameter values continuously adjust, allowing the system to converge to zero more rapidly.

Then, we divided the test set into two groups and assessed the performance using different methods. The results are presented in Table 2.

Table 2 shows that our proposed method outperforms other methods in terms of EMA, ERP, and ERFS. This indicates that our proposed method can better capture emotional states in visual aesthetics. In Dataset 1, the Machine Vision method consistently leads in accuracy, with 91.24% for EMA, 82.45% for ERP, and 92.22% for ERFS. This method significantly outperforms CNN and LSTM across all metrics. The CNN method shows lower accuracy, particularly in ERP (58.89%) and ERFS (60.02%), while LSTM yields moderate results with a peak accuracy of 73.24% for ERFS but remains below Machine Vision.

For Dataset 2, the Machine Vision method achieves the highest accuracy again, reaching 92.45% for EMA, 88.35% for ERP, and 96.22% for ERFS. The CNN method exhibits improved performance relative to Dataset 1, especially in EMA (80.21%) and ERFS (66.02%). The LSTM method also shows enhanced accuracy compared to Dataset 1, with 77.22% for EMA and 77.34% for ERP, though it still underperforms in ERFS with 61.22%.

To evaluate the robustness of our proposed method, we introduced varying degrees of interference to the dataset, such as adding irregular noise to visual images. The test results are illustrated in Table 3.

Table 3 shows that with the increase in interference levels, other methods experience a significant decrease in EMA, ERP, and ERFS. However, the proposed method in this article maintains good performance, indicating superior robustness. Machine Vision consistently leads in performance across all disturbance levels, with a slight decrease in accuracy as disturbance increases. At 10% disturbance, it achieves the highest accuracies (88.23% for EMA, 85.15% for ERP, 89.29% for ERFS). With 30% disturbance, accuracy drops slightly (85.34% for EMA, 86.19% for ERP, 88.39% for ERFS), and further decreases at 50% disturbance (82.45% for EMA, 83.35% for ERP, 86.22% for ERFS). This trend highlights Machine Vision’s robustness despite increasing disturbance.

CNN shows a significant drop in performance as disturbance increases. At 10% disturbance, CNN’s accuracies are relatively low compared to Machine Vision (67.13% for EMA, 52.11% for ERP, and 58.72% for ERFS). Accuracy further deteriorates with 30% disturbance (50.55% for EMA, 43.47% for ERP, 56.42% for ERFS) and 50% disturbance (40.25% for EMA, 43.44% for ERP, 46.52% for ERFS). This indicates that CNN is more sensitive to disturbance compared to other methods.

LSTM demonstrates mixed results, with moderate performance across disturbance levels. At 10% disturbance, LSTM’s accuracy is somewhat higher than CNN (60.24% for EMA, 62.45% for ERP, and 63.47% for ERFS). With 30% disturbance, LSTM improves ERP accuracy (77.34%) but shows lower performance in EMA and ERFS. At 50% disturbance, LSTM’s performance declines, especially for ERFS (41.42%), while EMA and ERP accuracies are relatively better compared to CNN at the same disturbance level.

Discussion and limitations

Potential applications

This method can be applied to sentiment analysis of media content, helping to understand and evaluate how visual content triggers emotional responses in the audience. For example, film and television production companies can use this method to assess the emotional expression of movies or videos, optimizing the plot and visual design. Emotion recognition can be used in personalized recommendation systems on social media and e-commerce platforms to recommend relevant content or products based on users’ emotional preferences. In education, emotional modeling can be used to evaluate students’ emotional responses to educational content and provide personalized emotional support and counseling.