An intelligent film recommender system based on emotional analysis

Emotional computing

The key of text-based emotion computing is to accurately understand natural language based on the analysis of the syntax and semantic structure of text sentences. Turney (2007) analyzed and calculated the similarity degree of words with the intention of introducing point mutual information, and reconstructed the composition of the dictionary by introducing polarity semantics, which increased the richness of part of speech structure in the dictionary. Chen & Yang (2013), based on the pre-set emotion dictionary, introduced LDA model tool to further extract the subject words in the dictionary, expanded the richness of emotional tendency, and provided more sufficient supplement for polysemy subject words of a word. Zhou, Yang & Yang (2013) introduced information entropy to identify the Chinese text information. The innovation lies in the use of the SO-PMI algorithm to further screen the emotional subject words in the text, and construct a new emotion dictionary for the new subject words. Li, Cao & Cao (2008) used the weight first extraction method to extract the head word, and integrated the traditional machine learning method with the weight priority to obtain the semantic trend of the text.

Affective computing from other modes (such as images and videos) has only recently begun to be considered (Bakshi, Kaur & Kaur, 2016). The purpose of multimedia emotion computing is to identify the audience’s emotion expected to be caused by a given stimulus (Zhao et al., 2017). One of the main challenges of multimedia emotion computing is the emotional gap, that is, the inconsistency between the features and the expected emotional state of users brought by perceptual signals (Hanjalic, 2006; Zhang, 2019). Perceptual inconsistencies make it insufficient to simply predict the major (average) emotion categories of highly subjective variables. Zhao et al. (2016) proposed that two multimodal emotional computing tasks can be performed to deal with subjective challenges: predicting the personalized emotional perception of each viewer and assigning multiple emotional tags to each stimulus. For the latter task, the multi label learning method can be used to assign multiple tags to each stimulus of equal importance, or predict the emotional distribution and try to learn the degree of each emotion (Yang, Sun & Sun, 2017).

The ambiguity and ambiguity of the semantic ambiguity of natural language and the multi-scale nature of the text content, especially the interaction of sarcasm, metaphor, politeness, linguistic and cultural characteristics and style, result in the critical technical challenge of accurately describing and defining the hybrid subdivision emotion. Most of the existing methods do not consider the timeliness of the user’s emotional attribute tags, so it is difficult to describe the dynamic emotional attributes of users.

Film recommendation

Collaborative filtering algorithm is the most commonly used film recommendation algorithm, but it relies on user historical data and has the problem of sparse data cold start. Therefore, many scholars have improved it. Jia & Yu (2019) proposed a modified score collaborative filtering algorithm, which modified the user score matrix through the confidence in user clustering information and association rule mining, and used the modified score method to replace the weighted calculation score algorithm, hence working on the exhibition of the recommendation algorithm. Yu et al. (2019) proposed a collaborative filtering algorithm to improve similarity. Considering whether the film has been seen or not, a collaborative filtering recommendation algorithm combining Pearson correlation coefficient and Jaccard algorithm is proposed with full consideration of the user scoring situation. Chen et al. (2018) thought that the existing recommendation algorithms, including shallow and deep methods, usually embed the user’s history into a single potential vector, which may have lost the correlation between the user’s history and future interest by item or feature level. In addition, Bayes, clustering, Horting, SVD and machine learning  (Breitfuss et al., 2021; Cintia Ganesha Putri, Leu & Seda, 2020) and other algorithms have also been applied to the recommendation system. Töscher, Jahrer & Bell (2009) proposed a hybrid recommendation algorithm, which combined different learning algorithms to achieve better accuracy. Bell & Koren (2007) improved the cosine measurement method in collaborative filtering algorithm, and used statistical user ratings to compare user rating data, thus enhancing the recommendation effect of collaborative filtering.

User emotion description model

The PAD three-dimensional emotional model is selected for user emotion modeling. Dimension emotion theory points out that emotion should be described in psychological dimension space. The theory benefits from expressing various emotions that are not necessarily described by specific emotion descriptors. As far as affective computing tasks are concerned, it is also a problem of the accuracy of discrete emotion classification, while dimensional emotion is the regression problem of continuous space. The emotional classification of film recommendation is shown in Fig. 1. The vertical axis is the activation dimension, which represents the intensity of emotion, and the horizontal axis is the quantity of potency and an evaluation of the degree of positive and negative emotions.

The emotion description model designed in this article is shown in Fig. 2. It defines the major categories of emotion description model. The subclass includes text emotion description and image emotion description, which can inherit the function of the parent class and add its own new functions.

Different symbols can be used to describe the user and alphanumerics can be used to enhance the display. Social media is the window of human emotion. A user’s historical data from Sina Weibo can be collected, and image and text data can also be extracted from Sina Weibo. These data can then be used to build a user sentiment tendency model.

Comment data contains a lot of image and text information, so the user-defined model based on comment data is constructed by combining text and image. For user u, we collect all original tweets, including posted photos, to form data set W.

Text and pictures in comment do not contain tag information and must be annotated automatically first. Suppose comment data package contains comment data w and image set $I_u=\left\{i_1,i_2,\dots ,i_n\right\}$, traverse all users’ comment data and obtain emotion attribute PAD vector. (1)${\displaystyle {\text{WeiboSenti}}_u=X_u=\left\{x_{u1},x_{u2},\dots ,x_{\mathrm{un}}\right\}.}$

In addition, film data contains rich pictures and text information, so the movie emotion model based on movie data is established by combining the text and pictures, that is, the emotion tendency value of the movie is calculated. For a movie m, we collect its published data set M containing titles, introductions, reviews, and images. Each element in M can be defined by title, introduction, comment, and corresponding photo set $I_m=\left\{i_1,i_2,\dots ,i_n\right\}$.

Because the text and pictures of the film do not contain any label information, it is necessary to label them first. For a film, the PAD feature vector of emotion distribution can be obtained through emotion analysis model. (2)${\displaystyle X_m=\left\{x_{m1},x_{m2},\dots ,x_{\mathrm{mn}}\right\}.}$

Image-text feature fusion based on PSO

Affective state transition

Affective state transition (AST) refers to the user’s change from one emotional state to another. AST is represented by a directed graph G(AS, DE), where AS is the set of emotional states and DE is the set of directed edges. AS_i → AS_j represents the AST from AS_i to AS_j, and $P\left(A{S}_i\to A{S}_j\right)$ represents the probability of the AST from AS_i to AS_j. Given a finite emotional state, the AST can be represented by the following matrix, whose element is the probability of the AST $\left\{P\left(A{S}_i\to A{S}_j\right)\right\}$. (9)${\displaystyle P{\left(AS\right)}_{M\times M}=\left(\begin{array}{cccc}\hfill A{S}_{1,1}\hfill & \hfill A{S}_{1,2}\hfill & \hfill \cdots \hfill & \hfill A{S}_{1,M}\hfill \\ \hfill A{S}_{2,1}\hfill & \hfill A{S}_{2,2}\hfill & \hfill \cdots \hfill & \hfill A{S}_{2,M}\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill A{S}_{M,1}\hfill & \hfill A{S}_{M,2}\hfill & \hfill \cdots \hfill & \hfill A{S}_{M,M}\hfill \end{array}\right)}$ where ${\sum }_{j=1}^{M}A{S}_{i,j}=1,$i =1 , …, M; AS_i,j ≠ AS_j,i. In the task of this article, 8 × 8 AST matrix $P{\left(AS\right)}_{8\times 8}=\left\{A{S}_{i,j}\mid 1\le i,j\le 8\right\}$, where 0 ≤ ES_i,j ≤ 1. The AST prior probability $P\left(A{S}_{i,j}\right)$ can be initialized, which represents the probability of self-loops with the same starting and ending emotional states. Emotional recommendation items identified from the user’s history list of comment.

Emotion is a complex physiological reaction of human beings, which is limited by many complex factors, so it is very difficult to predict the dynamic emotion of users. For the convenience of the study, the following hypothesis is given: according to the list of users’ comment history, the change of users’ emotional state is most likely to be caused. Given a history list of user u $W_u=\left\{w_1,w_2,\dots ,w_N\right\}$, the corresponding emotion sequence is represented by PAD emotion model $A{S}^u=\left\{A{S}_1^u,A{S}_2^u,\dots ,A{S}_N^u\right\}$, emotional sequence has influence on the excitation of continuous emotional state $A{S}_{N+i}^u$. According to the model, the influence of each comment in list W_u on current emotional state can be calculated.

The discriminant model directly learns the decision function f(x), that is, it models the mapping from input space to output space. Or directly to the distribution P (Y — X) model, namely the case markers in feature X Y appear probability, is the a posteriori probability. The discriminant and conditional probability theory are used to predict the dynamic emotional states. Under the condition of the observation value AS^u, the probability of the user’s current possible emotional states is calculated. Therefore, the influence probability of each emotional state in AS^u is given. (10)${\displaystyle P\left(A{S}_{j+k}^u\mid A{S}_j^u\right),1\le j\le N,1\le j+k\le N,\sum _{j=1}^{N}P\left(A{S}^u\mid A{S}_j^u\right)=1.}$

Calculation of emotion matching degree

Conditional Random Field (CRF) is the basic model of natural language processing. According to the comment history list, the CRF in Fig. 3 is used to predict the user’s emotional state.

The proposed scheme in this article is based on the prediction of users’ current emotional state. The emotional sequence obtained from users’ comment history list can not only predict users’ current emotional state, but also divide it into sub-sequences based on the same emotional state. Given a list of comment $W_u=\left\{w_1,w_2,\dots ,w_N\right\}$, k emotional states can be obtained through emotion recognition, then the sub-sequence represented by $W_u\left(A{S}_i\right)$ is selected as the recommendation basis, and the optimization ranking recommendation list with the highest emotion matching degree with $W_u\left(A{S}_i\right)$ is output.

According to the performance of emotion recognition, emotion features are output and mapped into PAD emotion space to form specific emotion representation. Emotional value of PAD can be represented as pad = A^Tℌ. Therefore, the emotion matching degree can be calculated by the following inner sum. (11)${\displaystyle \mathfrak{F}\left(w_{ui},m_c\right)=\left(A_{ui}^{\mathrm{T}}\mathfrak{H}\right){\left(A_c^{\mathrm{T}}\mathfrak{H}\right)}^{\mathrm{T}}=A_{ui}^{\mathrm{T}}\mathfrak{H}{\mathfrak{H}}^{\mathrm{T}}{A}_c}$ where, w_ui represents the emotional state of the predicted user, and m_c represents the emotional state of the c-th movie to be recommended. Given N recommended items, their emotional matching degree can be expressed by emotional similarity matrix, where each item $F_k\left(A{S}_i,A{S}^u\right)$ is non-negative.

Data set

In this article, a cross platform user sentiment preference modeling method is proposed. By extracting the personalized information of users, the overlapping users of the two platforms are modeled. The data of film comments in this article comes from IMDB dataset, and Film data from MovieLens dataset. Using the crawler technology, we obtained 2896 common users of Sina and Douban, downloaded the film list from Douban film website, including film ID, film title, film introduction, actor, director, etc., and the evaluation form includes user ID, film ID, score, comment and time stamp. A total of 8,273 reviews of 1,910 films by 2896 users where each user rates at least 20 movies.

Comparison of emotion feature recognition effect

In order to verify the effectiveness of the multi-modal fusion method, feature fusion and PSO optimized weighted feature fusion features are applied to PAD emotion recognition. In PSO algorithm, the initial population size is set to 20, and the inertia n = 0.5, learning factors 1 and 2 are 8 and 6, respectively

The feature weighted fusion emotion value recognition uses PSO algorithm iteration to find the optimal parameters, as shown in Fig. 4.

Figure 4 shows that when PSO algorithm learns weights, the best fitness value converges with the increase of iteration times. With the increase of iteration times, the fitness value decreases from the initial value, and tends to converge after eight iterations to obtain the global optimal value.

Comparison of different algorithms

The performance of the proposed algorithm is evaluated by comparing with several baseline methods. The results are shown in Fig. 5. Comparison methods include the following.

ItemKNN algorithm (Sarwar et al., 2001). In this method, the items to be recommended were clustered into several classes, and the average score of each class was used as the algorithm prediction score:

SVD++ algorithm (Koren, 2008). This method introduced implicit feedback on the basis of singular value decomposition (SVD);

Libfm (Rendle, 2012). This is a feature-based matrix decomposition. In the experiment, TF-IDF features are extracted from the items that users click on and candidate items, and the two types of features are connected as the input of LibFM;

Wide & deep (Cheng et al., 2016) used the combination of wide channel of linear transformation;

Deepfm (Guo et al., 2017) is a recommendation method combining decomposer and deep neural network, and the two parts use the same TF-IDF features;

DKN (Wang et al., 2018) learns the items to be recommended and the representation of users, where a set of embedding vectors is obtained for the historical items that users click on, and then the candidate items are automatically matched to each item by attention mechanism, and they are aggregated with different weights.

NCF (He et al., 2017) is a collaborative filtering based on neural network. which uses multi-layer perceptron and takes comment count information as input.

The model presented in this article significantly improves each of the evaluation indicators. Similarly, the improvement in the indicators of Recall@5 and MRR@5 was particularly marked. For the personalized recommendation system, the more advanced the recommendation movie is, the more it can capture the user’s attention. Therefore, the better effects of Recall@5 and MRR@5 reflect the better user satisfaction and experience brought by the personalized recommendation algorithm.

By integrating emotional information into the recommendation model to build a recommendation system, the results verify the feasibility of the proposed recommendation algorithm through multimodal emotion fusion. In terms of reflecting the recall performance of the recommendation system, the algorithm has achieved the best results, which shows that the algorithm can solve the personalized recommendation problem of the recommendation system based on multimodal emotion fusion to a certain extent.

Recommendation effect evaluation

To evaluate the recommendation results of the proposed film recommendation algorithm, the accuracy of film recommendation is evaluated according to the emotional value by measuring the integrity of the recommendation results and the target film. Through subjective evaluation, the recommendation results of this algorithm are compared with those of random selection method. Each user is randomly provided with the first five recommendations obtained from one of the two methods, and they independently choose the number of films they think match the film they want to watch.

The model proposed in this article can extract the emotions of users before watching movies, classify them, and recommend movies that are in line with the current emotions of users. Regardless of whether users are in positive or negative emotions, the successful recommendation accuracy is high. In addition, 25 historical information records of different ages were collected, and five movies were recommended for them to participate in model training (see Fig. 6). It can be seen that only 8% of people do not like the recommended films, indicating the effectiveness of this recommendation method.