Art appreciation model design based on improved PageRank and ECA-ResNeXt50 algorithm

PageRank algorithm

The PageRank algorithm has become a prevalent ranking algorithm that is widely employed in search scenarios where diverse datasets can be represented as graph structures, such as Web search (Roul & Sahoo, 2021), ER graph search (Chakrabarti, 2007), and keyword database search (Li et al., 2021a). The personalized PageRank algorithm inherits the principles of the classical PageRank algorithm and employs the data model (graph) link structure to calculate each node’s weight recursively. This algorithm simulates the user’s behavior of randomly visiting nodes in the graph by clicking on links, i.e., it follows a random walk model to calculate the probability of random visits to each node in the steady state. The personalized PageRank algorithm accounts for the static link structure between nodes when calculating node weights and the user preferences expressed in personalized information, such as user queries, favorite pages, and so on (Scozzafava et al., 2020).

The accuracy of existing PageRank algorithms is determined by the configuration of static parameters, such as the number of fingerprints and the selection of hub nodes, which cannot be dynamically adjusted at runtime (Gao, Yu & Zhang, 2020). Additionally, the algorithm’s running efficiency is directly determined by the precision requirements set during compilation and cannot be dynamically tuned. Nevertheless, since different users or applications have varying efficiency and accuracy demands for algorithms, estimation algorithms that support runtime tuning of efficiency and accuracy, such as incremental optimization, are imperative (Mo & Luo, 2021). Some scholars have proposed an improvement method based on user interest that involves collecting and analyzing user usage data to determine the direction of user interest, which can enhance the accuracy of recommended content (Roul & Sahoo, 2021). Another improvement approach is to start from page similarity, which is currently classified into two main categories (Lamurias, Ruas & Couto, 2019; Liu et al., 2019): one uses the space vector model to determine the similarity between the page and the linked page, assigning more weight to the page with greater similarity to solve the problem of average weights; the other is an improved method based on content filtering, which evaluates page text and HTML tags to make the query results more precise.

Visual sentiment analysis

Image sentiment analysis can be categorized into two approaches: visual features and semantic features (Xu et al., 2023). Zhu et al. (2022) defined 102 mid-level semantic representations for image sentiment analysis, resulting in better sentiment prediction results than visual low-level features alone. Zhao et al. (2019) proposed a sentiment analysis approach based on designing visual art sentiment subjects and corresponding visual art sentiment patterns based on different types of sentiment subjects. Other research has focused on learning the affective distribution of visual art patterns and further describing visual art characteristics. One feasible method is to calculate the overall sentiment value of an image based on the textual sentiment values of adjective-noun pairs and corresponding responses in the image. Jiang et al. (2020) constructed a strongly and weakly supervised coupled network system for visual sentiment differentiation of images by importing images into VGGNet, obtaining the entire image features from the fifth convolutional layer, and then using a spatial pooling strategy to obtain weights for each emotion type. Li et al. (2021b) proposed a 3D CNN combining the 3D Inception-ResNet layer and LSTM network to extract image spatial features using Inception ResNet and learn temporal relationships using LSTM, then apply this information for classification.

Text emotion feature extraction based on PageRank algorithm

Image emotion feature extraction based on improved ResNet

By incorporating the ECA mechanism into the SE-ResNet network design concept, we refine and enhance ResNeXt50. Following the convolutional layer within each residual unit, we introduce a batch normalization (BN) layer. This augmentation accelerates convergence and elevates the system’s training accuracy. The ECA mechanism is shown in Fig. 2.

Assuming that any of the feature transformations, including convolution, is denoted as Ftr. $\mathrm{X}\to \mathrm{U}$, where $\mathrm{X}\in {\mathrm{R}}^{\mathrm{H}\times \times \times \mathrm{c}},\mathrm{U}\in {\mathrm{R}}^{\mathrm{H}\times \parallel \times \mathrm{c}}$, then compress all global information in a channel descriptor using global averaging pooling (GAP). The spatial dimensionality $\mathrm{H}\times \mathrm{W}$ in shrinking feature $\mathrm{U}$ is used to generate statistical variables

(4) $Z_n=F_{sq}(u_n)=\frac{1}{H\times W}\sum _i^H\sum _j^W{u}_n(i,j)$where ${\mathrm{Z}}_{\mathrm{n}}$ can be interpreted as a collection of local features whose statistical information can express the whole image and have a global field of perception. The statistic Z does not require dimensionality reduction, and the following way can obtain the attention of each channel. The weight size is used as the measure of attention

(5) $$\rho =\sigma \left(W_k\cdot Z\right)$$where ${\mathrm{W}}_{\mathrm{k}}$ contains $\mathrm{k}\times \mathrm{C}$ parameters, which are defined as

(6) $$\left[\begin{array}{cccccccc}{\mathrm{w}}_1^1& \cdots & {\mathrm{w}}_1^{\mathrm{k}}& 0& 0& \cdots & \cdots & 0\\ 0& {\mathrm{w}}_2^2& \cdots & {\mathrm{w}}_2^{k+1}& 0& \cdots & \cdots & 0\\ ⋮& ⋮& ⋮& ⋮& \ddots & ⋮& ⋮& ⋮\\ 0& \cdots & 0& 0& \cdots & {\mathrm{w}}_c^{\mathrm{c}-\mathrm{k}+1}& \cdots & {\mathrm{w}}_{\mathrm{c}}\end{array}\right]$$where $\sigma$ represents the sigmoid nonlinear activation function.

In this article, 8 ResNeXt50 modules are stacked in series, and the ECA attention mechanism is embedded after each ResNeXt50 module to capture the interdependent associations between channels. It further enhances the image emotion feature extraction capability. The attention weights of each channel after the ECA module are

(7) $$\rho =\sigma \left(H^{\left(k\right)}\ast F_{sq}\left(Q\right)\right).$$

To prevent the degradation problem in deep networks, we introduce the concept of residuals to weight and add each channel to the original input features effectively.

(8) $$Y=\left(w\otimes Q\right)\oplus X$$where $\mathrm{Y}$ denotes the output feature map after a block. $\otimes$ represents the corresponding dot product of the elements. $\oplus$ represents the corresponding sum of elements.

By integrating the ECA mechanism in DCNN, the network can be tailored to meet the specific requirements of different depths. This module enhances the quality of extracted features at lower layers in shallow networks by highlighting informative features. In contrast, in deeper networks, the significance of this module becomes more pronounced as the extracted features become more strongly correlated with the target category.

Emotion feature integration

To combine emotional features from text and images, this article will use weight coefficients derived from an overarching emotional context within the statistical dual-mode framework to determine and select the most suitable emotional integration strategy. First, ${\mathrm{f}}_{\theta }\left({\mathrm{X}}^{\mathrm{i}}\right)$ is defined as the input features ${\mathrm{X}}^{\mathrm{i}}$ in the parameter $\theta$, estimate the probability distribution by the Sigmoid function with Eq. (9).

(9) $${\mathrm{f}}_{\theta }\left({\mathrm{X}}^{\mathrm{i}}\right)={\displaystyle \frac{1}{1+{\mathrm{e}}^{{\theta }^{\mathrm{T}}{\mathrm{X}}^{\mathrm{i}}}}.}$$

Equation (9) defines the degree of bimodal contribution to the overall sentiment. ${\rho }^{\mathrm{T}}$ and ${\rho }^{\mathrm{T}}$ the weights of the text and picture contributions to the overall sentiment are calculated as

(10) $$\begin{array}{cc}\hfill & \hfill {\rho }^T=f_{\theta }\left(X^T\right)-{\displaystyle \frac{1}{2}}\\ \hfill & \hfill {\rho }^I=f_{\theta }\left(X^I\right)-{\displaystyle \frac{1}{2}.}\end{array}$$

Then, the sentiment weight coefficients of the two modalities are compared to determine the appropriate fusion strategy, which is calculated as follows

(11) $$X^c=\{\begin{array}{cc}{X}^T\cup X^I& \mathrm{i}\mathrm{f}{\rho }^T\ast {\rho }^I>0\hfill \\ X^T& \mathrm{i}\mathrm{f}\left|{\rho }^T\right|-\left|{\rho }^I\right|\ge 0\\ X^I& \mathrm{i}\mathrm{f}\left|{\rho }^T\right|-\left|{\rho }^I\right|\le 0\end{array}$$where ${\mathrm{X}}^{\mathrm{c}}$ denotes the fused features. Sentiment classification is performed by fusing the features of the text and image modalities, provided that the product of their respective sentiment weights and sentiment weight coefficients is positive. Suppose the absolute difference between the sentiment weight coefficient values is greater than zero. In that case, the polarity of the sentiment is determined based on the sentiment weight coefficient of the text modality.

The cross-modal learning algorithm calculates the predicted probability distribution of image and text sentiment using Kullback-Leibler divergence. The sentiment probabilities of text and image are discrete events, assumed to be defined as Event A and Event B, respectively.

(12) $D_{KL}(A\parallel B)=\sum _i{P}_A(x_i)\mathrm{l}\mathrm{o}\mathrm{g}\left(\frac{P_A(x_i)}{P_B(x_i)}\right).$

The loss function of Sigmoid can be used to consider the loss between the expected estimate and the true label, as well as related to the loss between the fusion characteristics of the graphical features and the estimated distribution, as shown in Eq. (13).

(13) $$\begin{array}{rl}\mathrm{J}(\theta )=& \frac{1}{N}\sum _{i=1}^{N}D\{Y_i\parallel f_{{\theta }^c}(X_i^C)\}+\frac{\alpha }{2}{\theta }^T\theta \\ & +\frac{\beta }{N}\sum _{i=1}^N\left[D\right\{f_{{\theta }^C}(X_i^C)\parallel f_{{\theta }^T}(X_i^T)\left\}\right]+\left[D\right\{f_{{\theta }^C}(X_i^C)\parallel f_{{\theta }^I}(X_i^I)\left\}\right]\end{array}$$where $\theta =\left\{{\theta }^{\mathrm{c}},{\theta }^{\mathrm{T}},{\theta }^{\mathrm{I}}\right\}$ represents the cross-modal learning parameters. $\alpha$ and $\beta$ are the superparameters of the model. ${\displaystyle \frac{\alpha }{2}{\theta }^{\mathrm{T}}\theta }$ is the canonical term to prevent overfitting of the model.

Experimental setup

This article employs the hyperparameter method in the ResNeXt network to facilitate the sentiment classification task. The optimal kernel scale size for convolution is 3 × 3, while the quantization step is fixed at one. The optimal pooling size for the layer is 2 × 2, and the quantization step is two. The hyperparameters were determined via tuning of the pre-trained network using ResNeXt. The input images were normalized and have a maximum width of 256 × 256 pixels. Histogram equalization was used to achieve data enhancement.

Results and discussion

The model utilizes cross-entropy as the loss function and employs the Adam optimizer as the chosen optimization algorithm throughout the training process. Figure 3 presents the training outcomes.

The meticulous examination of the data reveals remarkable consistency in the model’s performance across the training and test sets. Surprisingly, there is a negligible disparity between the accuracy achieved on the test set and the training set, indicating the model’s robustness and ability to generalize effectively.

Furthermore, the unity also extends to the loss values, where the loss value observed in the test set closely mirrors that in the training set. This parallelism between the training and test set loss values underscores the model’s capacity to maintain a low error level when confronted with new, unseen data. This finding validates the model’s training process and suggests that it has successfully learned the underlying patterns and features present in the dataset, making it well-suited for practical applications across various domains. In essence, the model’s reliability and consistency in performance make it a promising candidate for real-world use cases, where generalizability and accuracy are paramount.

Model comparison

To evaluate the effectiveness of the proposed algorithm, a comparative test was conducted against several commonly used neural network models. These include:

(1) DCNN: This model utilizes one CNN to extract sentiment features from the text and image modalities separately and predicts the sentiment polarity of each modality. The outputs are then fused using an averaging strategy at the decision level.

(2) CCR: This model uses CNN to extract features from the image and caption text and then employs KL scatter to learn the consistent sentiment of both modalities. This model is a feature layer fusion approach.

(3) AttnFusion: This model uses BiLSTM to model video frame sequences and text sequences, and an attention mechanism is used to learn the alignment weights between video frames and text words. The features from both modalities are fused using this attention mechanism to produce a more accurate multi-channel feature representation. The aligned multi-peaked features are then fed into the sequence model for sentiment recognition.

(4) MDREA: This model employs two separate RNNs to independently encode data from image and text inputs. An attention vector is generated by computing the weight parameter between the image encoding vector and the text’s hidden state. The Softmax function is then applied to the vector to predict the sentiment category.

The performance evaluation of the sentiment analysis model described in “Experimental Setup” is presented in Figs. 4 and 5, in comparison with the art appreciation sentiment analysis model introduced in this article. The experimental assessment was conducted on two datasets obtained from the Flickr and Twitter websites, and the evaluation metrics employed were Precision, Recall, and F1.

The proposed model surpasses the other four models in precision, recall, and F1, as evident from the figure. The Flickr dataset’s accuracy improved by a substantial margin of 23.14%, 16.35%, 8.11%, and 4.26%, respectively, compared to the other four methods. Moreover, the proposed model exhibits superior robustness, even with smaller Twitter datasets. It outperforms the other four models, thereby establishing the better classification results of the proposed model in the field of cross-modal graphical and textual fusion sentiment analysis. By fusing text and image features, the model successfully leverages the correlation between different modalities, discovers deeper associations, and performs complementary sentiment information. This feature fusion approach also reduces discrepancies, proving its feasibility and effectiveness on cross-modal data.

Emotion classification results

Figures 6 and 7 present the confusion matrix results of the bimodal sentiment model. In the Flickr dataset, the model achieves an impressive probability of correct prediction for the four labels, namely “sad,” “happy,” “angry,” and “neutral,” with accuracy rates of 73.44%, 70.83%, 88.02%, and 59.38%, respectively. The Twitter dataset has a higher prediction accuracy, albeit with reduced recognition of the “happy” sentiment label. Overall, the model demonstrates the highest recognition rate for angry labels, while its performance is weakest for neutral labels, frequently misidentified as sad labels.

The experimental results above prove that the model proposed in this chapter performs exceptionally well across all evaluation indices. The proposed model achieves a high level of prediction accuracy for each sentiment label and outperforms both unimodal and other bimodal sentiment analysis models regarding classification performance and stability. Moreover, the proposed model exhibits remarkable generalization ability, further validating its effectiveness in sentiment analysis.

Discussion

To summarize, the proposed model in this article effectively improves sentiment classification accuracy and achieves the best sentiment recognition ability. Additionally, the model’s robustness and generalization ability have been significantly enhanced. Moreover, adding temporal feedback factors slightly improves the accuracy of the PageRank algorithm. This is because the temporal factor compensates for the PR value and ensures that new high-quality content is appropriately ranked. The ECA-based attention mechanism has also successfully improved classification accuracy while reducing the number of model parameters by enabling local cross-channel interaction of sentiment features through one-dimensional convolution. Furthermore, the model’s accuracy in distinguishing between positive and negative words differs significantly, largely due to the quality of seed word selection. Although the number of seed words with different sentiments can be equivalent, the quality of the seeds cannot be guaranteed. Nonetheless, the experimental results fully validate the feasibility of the proposed method. It is essential to emphasize that although the connection weights for graph nodes in the algorithm description are calculated using HowNet, other supervised or unsupervised methods can also be used to obtain the connection weights.

In specific application scenarios, through online interactive comments, teachers can capture the different emotions of individual learners towards the same artwork. With the increasing popularity of social media, many users share their current developments on these platforms, which often reflect their genuine emotions when posting messages. By analyzing such information, we can better understand the interests and preferences of users at a particular period and predict future trends based on user sentiment. This approach can also help in the development of art appreciation software, which includes image loading (PhotoLoader), text loading (TextLoader), and prediction results (ShowResult). The software’s structure is shown in Fig. 8.

The image loading function loads the user-uploaded image content into a designated variable, performs the necessary preprocessing steps, and uses the processed output as input for the prediction result class. Similarly, the text loading function loads the user’s input text content into a variable, performs the required preprocessing and sentiment score vector transformation processes, and uses the resulting output as input for the prediction result class. The primary function of the prediction result class is to combine the outputs from the image loading and text loading functions and employ them as inputs to the sentiment analysis algorithm, thereby accomplishing sentiment analysis of the fused graphical data.