Parametric art creation platform design based on visual delivery and multimedia data fusion

Traditional data fusion methods

Traditional methodologies for data fusion primarily encompass feature-level homology and decision-level fusion. Decision-level fusion involves amalgamating results obtained from the recognition of disparate data modalities, whereas feature-level fusion entails the amalgamation of features within the feature space post feature extraction and spatial transformation of distinct modality data. The culmination of feature fusion results in the derivation of ultimate recognition outcomes. In contrast to decision-level fusion, feature fusion adeptly exploits the synergies inherent in interconnecting different modal features within the feature space. Ngiam et al. (2011) introduced a dual-pathway self-encoder model, extending the self-encoder paradigm for the fusion of two modalities. This model intricately intertwines two self-encoders, each endowed with independent encoders and decoders, connected through a shared hidden representation layer. Leveraging an encoder grounded in the restricted Boltzmann machine (Lee, Ekanadham & Ng, 2007), this design facilitates efficient interaction of features from distinct modalities at the hidden representation layer. Wang et al. (2015) further refined this model by incorporating orthogonal regularization of the weight matrix, mitigating redundancy in information. Silberer & Lapata (2014) embraced the concept of dual-pathway self-encoders to extract high-dimensional features effectively from textual and image inputs. However, none of the aforementioned dual-path autocoder models have been deployed in the domain of multimodal emotion recognition tasks. The utilization of restricted Boltzmann machines in these models imposes limitations on feature extraction capabilities, rendering the training process cumbersome. In addition to autocoders, conventional correlation analysis (Hotelling, 1992) found application in modal fusion. Originally designed for measuring correlation between two vectors, its scope was limited. Andrew et al. (2013) innovatively proposed a deep typical correlation analysis algorithm, intertwining deep neural networks with typical correlation analysis. This integration enables the application of correlation analysis to high-dimensional features, leveraging deep neural networks to learn intricate nonlinear transformations from varied inputs. Subsequent work (Dumpala et al., 2018) harnessed the nonlinear transformation capabilities of deep typical correlation analysis, coupling it with feature correlation measures for applications in sentiment recognition. Beyond dual-pathway autocoders and deep typical correlation analysis, modal fusion has also explored the utilization of mutual information.

Research on deep multimodal feature layer fusion methods

In the relentless progression of deep learning technology, the prevailing approach in contemporary data analysis involves fusing data at distinct stages by extracting features through deep networks. This paradigm is exemplified by the utilization of advanced mechanisms such as attention mechanisms and generative adversarial networks to discern subtle correlations between modalities. For instance, Zadeh et al. (2018) introduced a pioneering memory fusion network, delineating modality-specific and cross-modality interactions across the temporal axis. This model employs a distinctive attention mechanism to generalize temporal information, incorporating multimodal gated memories. In a subsequent model (Zadeh et al., 2018), the same authors explored modal interaction within a time series using a “multi-attention module” within a neural network. The interactions are stored in a hybrid memory within a recurrent memory component. Additionally, Tsai et al. (2019) proposed a fusion model grounded in a multi-attention mechanism. This model leverages different modal data as query value, key value, and value in the attention module, uncovering cross-modal interaction information within the graph computation feature of the attention module. Sahu & Vechtomova (2021) innovated with a fusion algorithm grounded in generative adversarial networks. This approach reveals commonalities among diverse modal features through an adversarial training strategy.

In delving into the study of video data, an optimal approach involves fully harnessing the distinctive characteristics of the two modal information types: image and audio. Employing advanced feature layer fusion methods facilitates the precise analysis of data, particularly in the realm of multimodal video data encompassing both image and sound. In the exploration of multimodal data, with a primary emphasis on image data, Tran et al. (2015) introduced a novel approach involving the calculation of three-dimensional convolutional neural networks, considering spatial and temporal dimensions. This model, operating through successive frames and convolution operations, adeptly amalgamates temporal information within the video domain. Convolutional neural networks, known for their prowess in processing multi-channel image information, play a pivotal role in image recognition, segmentation, and detection, particularly in contexts such as human behavior recognition and video classification (Simonyan & Zisserman, 2014). The I3D algorithm (Carreira & Zisserman, 2017), grounded in the framework of a three-dimensional convolutional neural network, further refines classification efficiency and network performance by expanding its width. This augmentation enhances the algorithm’s proficiency in comprehending and classifying video data, showcasing notable advancements in the field.

The preceding investigation underscores the limitations inherent in data analysis reliant solely on single modal information. Optimal results are achieved through the fusion of multimodal information and the profound integration of network layers, thus elevating the efficacy of data analysis. Leveraging the capabilities of a 3D convolutional neural network for video features, particularly those with timestamped features, facilitates adept data analysis and feature extraction. Moreover, superior accuracy in analysis is attained through the fusion of multimodal features, specifically sound information, employing decision layer fusion. Therefore, a refined approach involves distinct feature extraction processes for image features and audio data. Subsequently, data analysis is elevated to a more sophisticated level through decision layer fusion, thereby achieving an enhanced and nuanced understanding of the underlying information.

I3D-based video feature extraction

The pivotal role of the three-dimensional convolutional neural network (3DCNN) cannot be overstated. Evolving from the foundations of the 2DCNN, the 3DCNN excels in capturing temporal information with heightened efficacy, enabling the effective utilization of temporal intricacies within video data (Al-Hammadi et al., 2020). In the context of video classification, the 3DCNN performs sliding window convolutions on inputs in three dimensions, encompassing both spatial and temporal dimensions. Within the structure of the 3DCNN, the 3D convolutional kernel manifests as a three-dimensional cube. The convolutional operation entails the superimposition of consecutive frames into a 3D sequence, followed by the computation of convolution between this sequence of video frames and the convolution kernel. The resultant feature maps derived from the convolutional layer are then associated with the input video frames, adeptly capturing the nuanced motion information embedded within the video. The 3D convolution operation mirrors its 2D counterpart but extends its application into both spatial and temporal dimensions.

For an input tensor X, a convolution kernel tensor W and bias b, the formulation of 3D convolution is articulated as follows:

(1) $${\mathrm{Y}}_{\mathrm{i},\mathrm{j},\mathrm{k}}={\sum }_{\mathrm{l},\mathrm{m},\mathrm{n}}{\mathrm{X}}_{\mathrm{i}+\mathrm{l},\mathrm{j}+\mathrm{m},\mathrm{k}+\mathrm{n}}\cdot {\mathrm{W}}_{\mathrm{l},\mathrm{m},\mathrm{n}}+\mathrm{b}$$where ${\mathrm{Y}}_{\mathrm{i},\mathrm{j},\mathrm{k}}$ are the elements of the output tensor, and $\mathrm{l},\mathrm{m},\mathrm{n}$ is the index of the convolution kernel in three dimensions. 3D pooling operations are commonly used to reduce the data size. Maximum pooling and average pooling are two common 3D pooling operations. For maximum pooling, the formula is as follows.

(2) $${\mathrm{Y}}_{\mathrm{i},\mathrm{j},\mathrm{k}}={\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathrm{l},\mathrm{m},\mathrm{n}}{\mathrm{X}}_{\mathrm{i}\times \mathrm{s}+\mathrm{l},\mathrm{j}\times \mathrm{s}+\mathrm{m},\mathrm{k}\times \mathrm{s}+\mathrm{n}}$$where ${\mathrm{Y}}_{\mathrm{i},\mathrm{j},\mathrm{k}}$ are the elements of the output tensor after pooling, and $\mathrm{s}$ is the step size of pooling.

The I3D convolutional neural network, a creation of Google DeepMind Labs, represents a distinctive 3D convolutional neural network derived from the “inflation” of the GoogleNet architecture. This augmentation involves the addition of batch normalization layers after each convolutional operation, mitigating the network’s convergence challenges and alleviating inherent issues related to internal input variability in convolutional neural networks.

In the temporal dimension design of the I3D convolutional neural network, careful consideration is given to preventing the merging of edge information from different objects when the sensory field of the temporal dimension exceeds that of the spatial dimension, and vice versa, ensuring the capture of dynamic scenes (Wang, Koniusz & Huynh, 2019). To achieve this, in the downsampling structure of the I3D network, the design of the first two pooling layers involves setting the step size of the temporal dimension to 1 and maintaining the step size of the spatial dimension at 2. Conversely, in the design of the last pooling layer, the step size of the temporal dimension is set to 2, while the step size of the spatial dimension is set to 7. For the remaining downsampling layers, the step size for both the temporal and spatial dimensions is set to 2. The detailed network structure is illustrated in Fig. 1.

In contrast to the C3D network, the I3D network incorporates a batch normalization layer into its structure. Leveraging the distinctive network design of GoogleNetV1, this inclusion effectively addresses issues of underfitting caused by excessively large parameters and mitigates training dispersion resulting from internal input variations. When compared to the P3D network, the I3D network showcases a uniquely crafted downsampling structure that enhances the efficiency of extracting feature information from videos. This results in a substantial improvement in the algorithm’s recognition accuracy. Furthermore, in comparison to the T3D network, the I3D network exhibits advantages in both recognition accuracy and speed. The asymmetric downsampling strategy in I3D—using a temporal stride of 1 and spatial stride of 2 in early layers—preserves fine-grained temporal information while reducing spatial redundancy. This design enhances the network’s sensitivity to motion dynamics without significantly increasing computational cost. Temporal downsampling is deferred to later stages to capture long-term dependencies, making this structure more effective than uniform stride configurations such as (2, 2, 2). In this work, we follow the inflation strategy proposed by Carreira & Zisserman (2017), where 2D convolutional filters pre-trained on ImageNet are inflated into 3D kernels by expanding their shape along the temporal dimension. Specifically, a 2D kernel of shape N × N is transformed into a 3D kernel of size T × N × N, where the temporal dimension is initialized either by repeating the 2D weights or scaling them to preserve the original filter response. This inflation mechanism allows the model to benefit from rich spatial representations learned on large-scale image datasets while effectively capturing temporal dynamics through end-to-end training on video data. This transfer learning approach accelerates convergence and improves generalization, particularly under limited video supervision.

MFCC-based audio feature extraction

Mel filtering emulates the nonlinear perception and filtering characteristics of the human auditory system on audio signals. The human auditory system exhibits selective attention to frequencies within an audio signal, with varying sensitivity across different frequency ranges. Notably, the human ear is more attuned to low-frequency signals, displaying enhanced receptivity to these frequencies compared to higher frequencies. The cochlea processes audio signals below 1,000 Hz in a linear fashion, while signals above 1,000 Hz are received in a logarithmic manner (Deng et al., 2020).

To replicate this characteristic, a method involves overlaying a triangular filter bank on the power spectrum. In addition to the cochlear filtering effect, the human ear subjectively and nonlinearly perceives sound. To mirror this subjective perception more closely to that of the human auditory system, the signal is measured using a subjective nonlinear scale known as the Mel scale. This entails converting the frequency domain signal to the Mel scale. The conversion relationship between the Mel scale and the frequency spectrum is illustrated below:

(3) $${\mathrm{f}}_{\mathrm{m}\mathrm{e}\mathrm{l}}=2595\ast \mathrm{l}\mathrm{o}{\mathrm{g}}_{10}\left(1+{\displaystyle \frac{\mathrm{f}}{700}}\right)$$

(4) $$\mathrm{f}=700\left({10}^{\frac{{\mathrm{f}}_{\mathrm{m}\mathrm{e}\mathrm{l}}}{2595}}-1\right).$$

Equation (3) is the frequency to Mel scale conversion equation, and Eq. (4) is the Mel scale to frequency conversion equation. In these equations, f is the linear frequency in Hertz, ${\mathrm{f}}_{\mathrm{m}\mathrm{e}\mathrm{l}}$ is the frequency in the Mel scale, which approximates human auditory perception of pitch.

The function of the Mel filter bank is to mimic the properties of the human ear in the perception of sound frequencies by converting the frequency axes in the spectrogram to the Mel scale.

(5) $${\mathrm{H}}_{\mathrm{m}}\left(\mathrm{k}\right)=\{\begin{array}{cc}0,\hfill & \mathrm{k}<\mathrm{f}\left(\mathrm{m}-1\right)\hfill \\ {\displaystyle \frac{\mathrm{k}-\mathrm{f}\left(\mathrm{m}-1\right)}{\mathrm{f}\left(\mathrm{m}\right)-\mathrm{f}\left(\mathrm{m}-1\right)},}\hfill & \mathrm{f}\left(\mathrm{m}-1\right)\le \mathrm{k}\le \mathrm{f}\left(\mathrm{m}\right)\hfill \\ {\displaystyle \frac{\mathrm{f}\left(\mathrm{m}+1\right)-\mathrm{k}}{\mathrm{f}\left(\mathrm{m}+1\right)-\mathrm{f}\left(\mathrm{m}\right)},}\hfill & \mathrm{f}\left(\mathrm{m}\right)\le \mathrm{k}\le \mathrm{f}\left(\mathrm{m}+1\right)\hfill \\ 0,\hfill & \mathrm{k}>\mathrm{f}\left(\mathrm{m}+1\right)\hfill \end{array}$$where $\mathrm{f}\left(\mathrm{m}\right)$ denotes the center frequency of the first $\mathrm{m}$ center frequency of the Meier filter. ${\mathrm{H}}_{\mathrm{m}}\left(\mathrm{k}\right)$ is the weighting coefficient of the m-th Mel filter at the k-th FFT. It defines the triangular filter shape applied to the spectrum.

The Meier filter bank accomplishes the work of smoothing the spectrum, highlighting the resonance peaks of the signal, and eliminating harmonics. At the same time, it reduces the amount of calculation. Because the human ear presents a logarithmic response to sound, the logarithmic energy value is calculated for the output after the power spectrum passes through the Meier filter bank. The logarithmic energy calculation formula is shown in Eq. (6).

(6) $$\mathrm{s}\left(\mathrm{m}\right)=\mathrm{l}\mathrm{n}\left(\sum _{\mathrm{k}=0}^{\mathrm{N}-1}{\left|{\mathrm{X}}_{\mathrm{a}}\left(\mathrm{k}\right)\right|}^2{\mathrm{H}}_{\mathrm{m}}\left(\mathrm{k}\right)\right)0\le \mathrm{m}\le \mathrm{M},$$where ${\mathrm{X}}_{\mathrm{a}}\left(\mathrm{k}\right)$ is the frequency $\mathrm{k}$ corresponding to the spectral energy value, obtained directly from the Fourier transform of the audio frame, and ${\mathrm{H}}_{\mathrm{m}}\left(\mathrm{k}\right)$ is the frequency $\mathrm{k}$ is the corresponding Meier energy value, obtained by filtering with a triangular filter. To balance the signal-to-noise ratio (SNR) of the sound signal spectrum, the features extracted by FBank can be normalized using mean normalization, where the average value of each frame is subtracted from all frames. Mean normalization after Meier filtering reduces the computational difficulty while maintaining feature invariance.

Cepstrum is a way of converting a frequency domain signal to the time domain. The cepstrum is obtained by first taking the logarithm of the amplitude value of the standard amplitude spectrum and then visualizing it. The process of obtaining the cepstrum using the discrete cosine transform is shown in Eq. (7):

(7) $$\mathrm{C}\left(\mathrm{n}\right)=\sum _{\mathrm{m}=0}^{\mathrm{N}-1}\mathrm{s}\left(\mathrm{m}\right)\ast \mathrm{c}\mathrm{o}\mathrm{s}\left({\displaystyle \frac{\mathrm{\pi }\mathrm{n}\left(\mathrm{m}-0.5\right)}{\mathrm{M}}}\right),\mathrm{n}=1,2,\dots ,\mathrm{L}$$where $\mathrm{C}\left(\mathrm{n}\right)$ is the n-th MFCC coefficient, $\mathrm{L}$ denotes order of coefficients, and $\mathrm{M}$ is the number of diagonal filters, and $\mathrm{s}\left(\mathrm{m}\right)$ denotes the logarithmic energy value of the output of the first $\mathrm{m}$ is the number of diagonal filters, and denotes the logarithmic energy value of the output of the first diagonal filter, which is obtained from Eq. (7). This is the standard type-II DCT used in MFCC extraction, which provides energy compaction and decorrelation, and ensures compatibility with prior works in speech and audio processing. The waveform of the cepstrum is similar to the waveform of the spectrum, so when the low frequency in the spectrum has a higher energy value, the low-frequency cepstrum coefficients will also obtain a higher value, and when there are peaks in the spectrum of the high-frequency peaks, the high-frequency cepstrum will also appear peaks. In the spectrum of an audio signal, the frequency corresponding to the higher peak is the main frequency of the audio signal, and the peak is the resonance peak. The main peaks of the audio signal are its characteristics, and the similarity between the spectrum and the cepstrum waveform makes this feature retained. The above process of obtaining the MFCC feature can be represented in Fig. 2 (Gao, 2023).

Bimodal time series classification model for the audio and image feature fusion

Following the completion of feature extraction utilizing the I3D network for image features and employing MFCC for audio feature extraction, a convergence at the decision level is executed. Subsequently, we introduce the BTSCM network model, designed for the integrated classification of multimedia data. The schematic representation of the BTSCM network model and the procedural framework for feature fusion is illustrated in Fig. 3.

The BTSCM network, as proposed, comprises three primary components: a module for dimensionality reduction of features based on a fully connected network (Lu et al., 2019), a module dedicated to enhancing time series features through LSTM (Shi et al., 2022), and a model augmentation segment based on reinforcement learning using the deep Q network (DQN). The specific amalgamation is depicted in the rightmost section of Fig. 3. Initially, the model integrates with features derived from the extraction network. Subsequently, data undergoes feature extraction and dimensionality reduction via the FCN network. Following this, the enhancement of data is accomplished through the utilization of the LSTM network in conjunction with reinforcement learning, ensuring precise categorization of the data. The subsequent sections will elaborate on each of the three modules.

(1) FCN module: FCN replaces the traditional fully-connected layer with a fully-convolutional layer, enabling the network to accept input images of any size and output predictions of the corresponding size. This enables FCN to process each pixel in an image rather than just classifying the entire image. FCN captures semantic information of an image at multiple scales by integrating different levels of feature maps. This makes the model better adaptable for targets of different sizes and shapes. For the 1DFCN used in this article can be computed by Eq. (8)

(8) $${\hat{\mathrm{E}}}_{\mathrm{i},\mathrm{t}}^{\left(\mathrm{l}\right)}=\mathrm{f}\left({\mathrm{b}}_{\mathrm{i}}^{\left(\mathrm{l}\right)}+\sum _{{\mathrm{t}}^{\mathrm{\prime }}=1}^{\mathrm{d}}⟨{\mathrm{W}}_{\mathrm{i},{\mathrm{t}}^{\mathrm{\prime }},.}^{\left(\mathrm{l}\right)},{\mathrm{E}}_{.,\mathrm{t}+\mathrm{d}-{\mathrm{t}}^{\mathrm{\prime }}}^{\left(\mathrm{l}-1\right)}⟩\right)$$where ${\mathrm{W}}^{\left(\mathrm{l}\right)}$ and ${\mathrm{b}}^{\left(\mathrm{l}\right)}$ are the weight and biases ${\mathrm{E}}_{\mathrm{t}}^{\mathrm{l}}$ is a function of the incoming activation matrix $\mathrm{f}(\cdot )$ is a rectified linear unit. In this article we have chosen FCN32s to accomplish the dimensionality reduction of the data. K is the kernel half-width, r is the dilation rate. This convolution is applied along the temporal dimension, while treating the feature dimension as the input channels. This design enables the 1D-FCN to model both local and long-range temporal patterns depending on the dilation rate. In our implementation, the 1D-FCN is applied along the temporal dimension of the concatenated feature sequence. Specifically, for each video, we first concatenate multi-modal features at the frame level, resulting in a sequence of fixed-length feature vectors over time. The 1D convolutions then operate with kernels that slide along the time axis, while treating the feature dimension as the input channel dimension. This design allows the network to learn temporal dependencies and local patterns across consecutive frames, without mixing information across feature channels.

In this study, we selected FCN32s primarily due to its efficiency in reducing the temporal resolution while retaining high-level semantic representations. FCN32s applies a larger stride in its decoding path, which allows for faster computation and simpler integration with the sequential modeling components (e.g., LSTM). Given the goal of this work is to analyze multi-modal feature interactions rather than fine-grained temporal localization, FCN32s offers a practical balance between performance and complexity. Although variants like FCN8s or FCN16s could provide finer temporal resolution, they significantly increase computational overhead and are more suitable for tasks requiring dense, frame-level predictions. We leave the comparative study of different FCN depths as part of future work focused on architectural optimization.

(2) DQN module: Before the spatio-temporal feature extraction and analysis of LSTM, we add a reinforcement learning module, and in this article, we realize the reinforcement of the model through the DQN network. DQN is a kind of deep reinforcement learning model, which is used for solving the reinforcement learning problem in the discrete action space. It combines the ideas of deep learning and Q-learning, and aims to learn to predict optimal actions in complex environments through neural networks. Reinforcement learning mainly searches for the optimal Q-value through the following Formula (9) to optimize the model.

(9) $$\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right)=\left(1-\mathrm{\alpha }\right)\cdot \mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right)+\mathrm{\alpha }\cdot ({\mathrm{r}}_{\mathrm{t}}+\mathrm{\gamma }\cdot \underset{\mathrm{a}}{\mathrm{m}\mathrm{a}\mathrm{x}}\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}+1},\mathrm{a}\right))$$where $\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right)$ denotes the state in which ${\mathrm{s}}_{\mathrm{t}}$ action is taken in the state ${\mathrm{a}}_{\mathrm{t}}$ of the $\mathrm{Q}$ value, and ${\mathrm{r}}_{\mathrm{t}}$ is the value of the action taken in the state ${\mathrm{s}}_{\mathrm{t}}$ is the value of the immediate reward obtained after taking an action in the state ${\mathrm{a}}_{\mathrm{t}}$ the immediate reward obtained after the action is taken, the ${\mathrm{s}}_{\mathrm{t}+1}$ is the next state transferred to, the $\mathrm{\alpha }$ is the learning rate, the $\mathrm{\gamma }$ is the discount factor, and $\mathrm{m}\mathrm{a}{\mathrm{x}}_{\mathrm{a}}\mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}+1},\mathrm{a}\right)$ denotes the next state in which the ${\mathrm{s}}_{\mathrm{t}+1}$ in which the action is selected $\mathrm{a}$ of the maximum Q value.

(3) LSTM module: LSTM is specially designed to solve the long sequence dependence problem, which contains an input gate, a forgetting gate, and an output gate, and through the control of the three gates, it can be a good solution to the gradient explosion problem, and significantly improve the performance of the model. The computational process of its main three gates is shown in Eqs. (10)–(12):

(10) $${\mathrm{i}}_{\mathrm{t}}=\mathrm{\sigma }\left({\mathrm{W}}_{\mathrm{i}\mathrm{i}}{\mathrm{x}}_{\mathrm{t}}+{\mathrm{b}}_{\mathrm{i}\mathrm{i}}+{\mathrm{W}}_{\mathrm{h}\mathrm{i}}{\mathrm{h}}_{\mathrm{t}-1}+{\mathrm{b}}_{\mathrm{h}\mathrm{i}}\right)$$

(11) $${\mathrm{f}}_{\mathrm{t}}=\mathrm{\sigma }\left({\mathrm{W}}_{\mathrm{i}\mathrm{f}}{\mathrm{x}}_{\mathrm{t}}+{\mathrm{b}}_{\mathrm{i}\mathrm{f}}+{\mathrm{W}}_{\mathrm{h}\mathrm{f}}{\mathrm{h}}_{\mathrm{t}-1}+{\mathrm{b}}_{\mathrm{h}\mathrm{f}}\right)$$

(12) $${\mathrm{o}}_{\mathrm{t}}=\mathrm{\sigma }\left({\mathrm{W}}_{\mathrm{i}\mathrm{o}}{\mathrm{x}}_{\mathrm{t}}+{\mathrm{b}}_{\mathrm{i}\mathrm{o}}+{\mathrm{W}}_{\mathrm{h}\mathrm{o}}{\mathrm{h}}_{\mathrm{t}-1}+{\mathrm{b}}_{\mathrm{h}\mathrm{o}}\right)$$where ${\mathrm{x}}_{\mathrm{t}}$ is the input, and ${\mathrm{h}}_{\mathrm{t}-1}$ is the previous hidden state, and $\mathrm{\sigma }$ denotes the activation function, and $\mathrm{W}$ and $\mathrm{b}$ are the weights and biases, respectively. We further define the cell state update and the hidden state computation, which are essential components of the LSTM unit. The equations are as follows:

(13) $$c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$$

(14) $$c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t$$

(15) $$h_t=o_t\odot \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(c_t\right)$$where ${\tilde{c}}_t$ is the candidate cell state, $c_t$ is the updated cell state, $h_t$ is the hidden state (also the LSTM output at time step $t$), $\odot$ denotes element-wise multiplication, tanh is the hyperbolic tangent activation function. These equations define how information is selectively retained, updated, and exposed at each time step, ensuring long-range temporal dependencies are effectively modeled.

In this article we choose a two-layer LSTM network to help complete the data processing and add a fully connected layer to its subsequent to complete the classification. In this article we choose the cross entropy function as the loss function for model tuning.

After further processing of the model features, we add a classification layer after the BTSCM module, which mainly consists of dropout and softmax functions to complete the final classification. In our framework, the DQN serves as a high-level decision-making module that dynamically interacts with the sequential learning process. Specifically, the “action” in the DQN refers to the selection or modulation of attention over feature representations at each time step, effectively guiding the model on which temporal segments or modality features to prioritize. This reinforcement-based mechanism operates in parallel with the LSTM and influences its memory updates by providing external signals that emphasize salient information and suppress redundant or noisy inputs. As a result, the LSTM benefits from a more focused and adaptive encoding of long-term dependencies, thereby improving overall classification accuracy. This integration enables a closed-loop structure where DQN continuously refines the temporal focus based on feedback from the classification performance.

Dataset

Considering the paramount objective of discerning diverse human movement conditions within multimedia data pertaining to art and design, this article conducts its analysis utilizing four databases encompassing human movement video data along with associated audio information: UCF101, Kinetics, Sports-1M, and ActivityNet. UCF101, a widely acknowledged video classification dataset, features 101 distinct action categories, amounting to 13,320 video clips. Kinetics-400, a comprehensive video classification dataset, comprises 400 diverse action categories, totaling nearly 300,000 video clips, rendering it one of the most expansive publicly accessible repositories for video categorization. Sports-1M, an extensive dataset, encompasses 1,133,514 YouTube videos spanning 487 distinct movement categories. ActivityNet, a large-scale video comprehension dataset, encapsulates 203 varied activity categories sourced from YouTube, resulting in a compilation of 10,024 video clips. Despite the expansive nature of these datasets, this article selectively employs a subset for experimentation, owing to constraints imposed by experimental equipment. Within these datasets, video clips commonly consist of sequences of video frames accompanied by corresponding audio clips, aligning seamlessly with the model employed in this study. Throughout the model application process, we initialize the video and undertake a comprehensive analysis of the pertinent data. For all datasets, only video clips containing synchronized audio and clean action labels were retained. To ensure class balance and reduce bias, we manually selected equal numbers of samples per class where feasible. Annotations were taken from the official dataset labels, and no additional manual labeling was performed. This curated subset ensures a manageable yet representative sample for evaluating the system’s performance in multimedia art contexts.

Experiment setup and details

After determining the dataset, we determined the evaluation method of the model, and considering to carry out the classification problem research, we used the classical precision, recall and F1-score metrics for model evaluation, which are defined as shown in Eqs. (16)–(18):

(16) $$P={\displaystyle \frac{TP}{TP+FP}}$$

(17) $$R={\displaystyle \frac{TP}{TP+FN}}$$

(18) $$F1={\displaystyle \frac{2\times P\times R}{P+R}}$$where P denotes precision; R denotes recall; F1 denotes F1 coefficient; TP denotes true case; FP denotes false positive case; FN denotes false negative case. Based on the above data and evaluation indexes we carried out the training of the model, and its specific process is shown in Algorithm 1.

To enhance the evaluative process of the models, we have curated a selection of network methodologies capable of integrating video image features and audio features for comparative analysis. The methodologies under consideration include: tensor fusion network (TFN): TFN adeptly models interactions across unimodal, bimodal, and trimodal domains through a three-dimensional Cartesian product, facilitating a sophisticated fusion of feature spaces (Zadeh et al., 2017). Knowledge enriched transformer (KET): The KET model intricately models contextual information employing a hierarchical self-attention module and dynamically introduces external general knowledge into the information flow via a context-aware affective-attention mechanism (Zhong, Wang & Miao, 2019). COSMIC: The COSMIC framework, proposed in this work, is grounded in a general knowledge-oriented approach and relies on an extensive knowledge base, effectively capturing audio features (Ghosal et al., 2020). Bidirectional emotional recurrent unit (BiERU): The BiERU model represents an exceedingly parameter-efficient framework that disregards participant influence. Leveraging a generalized neural tensor block and a two-channel feature extractor, it adeptly captures contextual information (Li et al., 2022).

Experiment results and analysis

Following the culmination of model training, a comprehensive comparative analysis of results across diverse datasets transpired. Deliberating on practical application scenarios and data accessibility, this article meticulously curated a subset of 50 actions from each database for in-depth analysis, culminating in a final classification task. To ensure fair evaluation and avoid class imbalance, the 50 selected actions were chosen to represent a diverse range of motion types (e.g., dynamic, subtle, upper/lower body), while maintaining an approximately equal number of samples per class within each dataset. Selection was guided by the availability of synchronized audio-visual data and the clarity of action labels. This strategy minimizes potential bias in evaluation metrics and ensures comparability across datasets. The outcomes pertaining to the UCF101 dataset are meticulously delineated in Table 1 and visually depicted in Fig. 4.

The discernment across the three indicators readily reveals the commendable recognition efficacy of the methods introduced in this article. The model exhibits a nuanced equilibrium in performance. It is noteworthy, however, that the superior quality and reduced noise within the dataset contribute to the heightened results observed in multi-classification tasks. For a more expansive evaluation, the outcomes pertaining to the other two datasets, Kinetics and Sports-1M—widely employed in the multi-classification study of human motor behavior—are meticulously presented in Tables 2 and 3, complemented by the graphical representation in Figs. 5 and 6.

Given the extensive diversity within the dataset and the current state-of-the-art (SOTA) algorithms struggling to surpass a classification performance threshold of 70%, it is noteworthy that the recognition rate achieved by the method proposed in this article, while lower, still surpasses that of existing multimodal fusion methods. Figure 6 illustrates that the Recall of the Bi-ERU method marginally surpasses that of the proposed method, yet the overall performance disparity is minimal. This observation substantiates the assertion that the proposed method exhibits a more balanced performance. The comparison result on the ActivityNet dataset is shown in Table 4 and Fig. 7.

Building upon this foundation, an analysis was conducted on the ActivityNet dataset. Given the dataset’s clarity, the overall performance surpasses that of the aforementioned two datasets, with precision reaching 84.7%, notably outperforming other methods. Building upon this success, an analysis of data from the forthcoming art platform was undertaken. The platform’s data stream, characterized by a lower volume, predominantly comprises uploaded short videos and related audio. Selecting pertinent clips for testing, the results for various multimodal data analysis methods are vividly illustrated in Fig. 8.

As shown in Fig. 8, we still compared four types of networks fusing audio and image information such as TFN, and the results showed that all types of methods achieved more than 85% precision, while the network proposed in this article achieved better results under all three types of indexes, and its results showed that the method proposed in this article has a better equilibrium performance, and improves the recognition accuracy to a certain extent. After completing the model analysis, we carried out the ablation experiment of the reinforcement learning module, and its results are shown in Table 5 and Fig. 9.

The mean precision, recall and F1-score and the related standard deviation for reinforcement learning influence is also represented as follows:

In the conducted ablation experiments, the model underwent testing both with and without the incorporation of reinforcement learning on the self-established dataset. The experimental outcomes reveal that the inclusion of the reinforcement learning module enhances the recognition efficacy of the LSTM feature extraction layer employed in this study. Across various batch sizes, the effect observed in the model with the reinforcement learning module surpasses that of the model without, affirming the robustness of the BTSCM method and its ability to achieve superior recognition outcomes.