Music emotion representation based on non-negative matrix factorization algorithm and user label information

Emotional representation of music

At first, music emotion classification is based on audio features, but the effect of music emotion recognition only based on audio features is not ideal. With the development of multimodal information fusion technology, more and more scholars use the idea of multimodality to retrieve music information. “Multimodal fusion” is the combination of different information sources and helps the system to complete specific tasks through the collaborative use of information provided by multiple modes (Katsaggelos, Bahaadini & Molina, 2015). Researchers have shown that the fusion of audio and lyrics can improve classification accuracy (Laurier, Grivolla & Herrera, 2008; Yang et al., 2008; Yang & Lee, 2004). Jamdar et al. (2015) and Wang et al. (2020) used keyword discovery technology in the process of lyrics processing, and extracted melody MFCC and pitch features. Experimental results show that the combination of the two features can significantly improve the performance. Laurier, Grivolla & Herrera (2008) used language model differences to select words, moreover, the lyrics were characterized by bag of words and combined with acoustic features to improve the accuracy of emotion classification; Yang et al. (2008) transformed lyrics into 182 psychological categories, and the fusion features improved the classification effect.

User emotion recognition

In the field of music information retrieval, more and more researchers combine music emotion with user emotion to carry out research. Music is not only emotional in itself, but also evokes the emotions of the audience (Qianwen, 2022). The ability of music to induce emotions ensures its prominent position in human culture and daily life. Many studies have proposed a variety of methods to classify and identify users’ emotional states. Kiritchenko, Zhu & Mohammad (2014) constructed an emotion analysis system to identify emotions in informal texts such as tweets and messages, and achieved excellent results on SMS test set and film review corpus; Godbole, Srinivasaiah & Skiena (2007) tested public sentiment for news, track the evolution of public sentiment index and support rate for the president; Saari & Eerola (2014) uses social tags to identify music audience emotions. In addition, many researchers use dictionary based methods to carry out research. For example, Wilson, Wiebe & Hoffmann (2005) constructed the subjective vocabulary by questioning the respondents, and finally got about 8,000 emotional tags; The NRC emotion lexicon constructed by Mohammad & Turney (2010) through Mechanical Turk has about 14,000 words of emotion and tag. Machine learning can effectively solve the problem of emotion dictionary, but it cannot solve the problem of unknown words. Lixing, Ming & Maosong (2012) proposed a method for sentiment classification of Twitter using word vector and realized effective supervision of the emotional polarity of text with the help of three neural networks.

Music emotion representation model based on NMF

Non-negative matrix decomposition (NMF) combines cryptic meaning and machine learning characteristics, which can dig deeper connections between users and items. Therefore, the prediction accuracy is relatively high, and the prediction accuracy is higher than neighborhood based collaborative filtering and content-based recommendation algorithm. The model can be trained by stochastic gradient descent method and alternating least square method. At the same time, matrix decomposition has low time and space complexity. The mapping of high-dimensional matrix into two low-dimensional matrices saves storage space, and the training process is time-consuming, but can be completed offline. Score prediction is generally calculated online, directly using the parameters obtained from offline training, which can be recommended in real time.

NMF model

The NMF technology is specifically explained. The main problem of the model is transformed into the structure shown in Fig. 1: construct a music feature matrix, use NMF to map the song list and music to the emotional space at the same time, and obtain the representation or distribution of the song list and music in the emotional space.

The co-occurrence information is represented as a two-dimensional matrix. This type of matrix is a grid-like structure with rows and columns, forming a table. In this context, the rows correspond to different song lists, and the columns correspond to individual musical pieces. The elements of the matrix represent the co-occurrence strength or count of each musical piece in each song list. For example, if element (i, j) of the matrix is 3, it means that the j-th musical piece appears three times in the i-th song list.

By using this two-dimensional matrix to represent the co-occurrence information, researchers, data analysts, or music enthusiasts can analyze patterns and relationships between different song lists and musical pieces. It allows them to discover which songs are commonly associated with specific playlists or which playlists share similar songs, thus aiding in music recommendation systems, playlist generation, and other music-related data analyses. The co-occurrence information of playlist and music is represented as a two-dimensional matrix, in which each row represents a playlist and each column represents a piece of music. The value in the matrix is 0 or 1, indicating whether the corresponding music appears in the playlist (1 for the occurrence, 0 for the absence). Due to the large number of music in the list, there will be problems such as excessive resource consumption and time-consuming in data calculation. The form of matrix X should be preprocessed as follows:

(1)${\displaystyle X=\left(\begin{array}{ccc}\hfill a_{1}1\hfill & \hfill \cdots \hfill & \hfill a_{1n}\hfill \\ \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill a_{m1}\hfill & \hfill \cdots \hfill & \hfill a_{mn}\hfill \end{array}\right)}$

where m represents the singular number of songs and n represents the number of music

A two non-negative factorization matrices of the X should be arranged to replace U and V, which make the original matrix X as close as possible to the factorized result UV^T. as shown in Formula (2):

(2)${\displaystyle \begin{array}{c}\hfill O={∥X-U{V}^T∥}_F^2\hfill \\ \hfill \mathrm{ s}.\mathrm{t }U\ge 0,V\ge 0\hfill \end{array}}$

Emotional labels

In the context of music emotion representation, the song sheet or music emotional label information which is labeled by experts is included in the matrix decomposition as external information. Assuming that the following objective functions should be minimized as follows:

(3)${\displaystyle {∥G_v\left(V-V_0\right)∥}_F^2.}$

Among them, V₀ represents music-emotion label matrix, G_v is the diagonal indicator matrix representing the musical level. G_v (i, i) = 1 indicates that the I track contains an emotional indicator, and G_v (i, i) = 0 if it does not.

Emotional representation

The quintessence of music feeling acknowledgment is the portrayal of music in the profound space. Let the emotional distribution of music i be V_i, then music i can be expressed in the k-dimensional emotional space as:

(4)${\displaystyle V_i=\left(V_{i1},V_{i2},\dots ,V_{ik}\right).}$

The V_i corresponding to music i is normalized, and the emotional probability distribution $s_i^{\mathrm{\ast }}$ corresponding to music i is obtained after normalization. The calculation formula is as follows:

(5)${\displaystyle s_i^{\mathrm{\ast }}=\left(\frac{V_{ij}}{{\sum }_{j=1}^k{V}_{ij}}\right)}$

where k represents the dimension of emotional space.

For each piece of music, the highest emotion category is taken. The calculation formula is as follows:

(6)${\displaystyle e_i^{\mathrm{\ast }}\leftarrow argmax\left(s_i^{\mathrm{\ast }}.\right)}$

Emotional computing

Through the above calculation, the distance relationship between music and music is obtained. The first k music most similar to music i is regarded as the neighbor of music i, which is represented by N(i). Assume that the emotion representation vector of unknown music i is s_i, and the emotion distribution vector of the j-th neighbor of music i is $s_j^{\mathrm{\ast }}$. Use the average value of the emotion vector of neighbor N(i) of music i to represent the emotion of unknown music i, then the calculation formula of the emotion representation vector of music i is as follows:

(7)${\displaystyle s_i^{\mathrm{\ast }}=\frac{{\sum }_{i=1}^{N\left(i\right)}{s}_j^{\mathrm{\ast }}}{{\sum }_{j=1}^{N\left(i\right)}{s}_j^{\mathrm{\ast }}}.}$

Finally, the emotional category of music with the maximum value is selected as the result of emotional classification of music, and the emotional category to which music i ultimately belongs is s $e_i^{\mathrm{\ast }}$, which can be calculated as follows:

(8)${\displaystyle e_i^{\mathrm{\ast }}\leftarrow argmax\left(s_i^{\mathrm{\ast }}\right).}$

Overall framework

The music that users have listened to forms the music list. In order to realize the recognition of users’ immediate emotions, the emotion of each music in the music list should be identified first. Combined with the user’s historical listening behavior, an emotion recognition algorithm is designed to calculate the user’s emotional perception when listening to songs. Finally, the emotion of users listening to songs is classified into “cure, relaxing, romantic, nostalgic, excited, lonely and quiet”. Figure 2 shows the framework of user emotion recognition.

The proposed framework encompasses three main components: Melody Compilation, Sentiment Elicitation, and Affective Computation. Firstly, in the Melody Compilation phase, it is essential to acquire historical user data from their music-listening experiences. Generally, users create a song compilation, serving as a curated list of songs. Secondly, in the Sentiment Elicitation phase, a specialized algorithm for music sentiment extraction is designed to capture all emotions embedded in the song compilation, whether overt or latent. Lastly, during the Affective Computation stage, the acquired music sentiments are utilized to compute the user’s emotional representation based on their music compilation. This computation yields distinct emotional categories, thereby accomplishing the objective of emotion recognition. Ultimately, the outcomes of user emotion recognition can be applied to various specific scenarios.

Algorithm design

First, the user emotion recognition is transformed into a mathematical problem. Let the emotion distribution vector of the i-th user be $u_i^{\mathrm{\ast }}$, and there are k classes user emotion at present, and the emotion of user i can be expressed as:

(9)${\displaystyle u_i=\left(u_i^1,\dots ,u_i^j,..,u_i^k\right)}$

Based on the user’s listening list, the number of times the user listens to songs is taken as the weight, and the weighted average value of the emotional vector of music in the historical listening list represents the user’s emotions, and the user’s emotional distribution is finally obtained. The corresponding weight of song j in the listening list is set as w_j, and the weight calculation method is as follows: If music ranks first, that is, the number of times users listen to songs is the largest, then the weight of music is 100; While if the music is not ranked first, the corresponding weight = the number of times the music is listened to/the times music ranked first is listened to ×100

Assuming that the distribution vector of music emotion is $s_j^{\mathrm{\ast }}$, and the emotion of music can be expressed as:

(10)${\displaystyle s_j=\left(s_j^1,\dots ,s_j^k\right)}$

The calculation formula of user i’s emotional representation is as follows:

(11)${\displaystyle u_i^{\mathrm{\ast }}=\frac{{\sum }_{j=1}^k{w}_j\mathrm{\ast }{s}_j^{\mathrm{\ast }}}{{\sum }_{j=1}^k{w}_j\mathrm{\ast }{s}_j^{\mathrm{\ast }}}}$

After obtaining the emotional distribution vector of the user, the emotional dimension of the maximum value is taken as the final emotion of the user. The final emotional class of user i is expressed as follows:

(12)${\displaystyle e_i^{\mathrm{\ast }}\leftarrow argmax\left(u_i^{\mathrm{\ast }}\right).}$

Data set

The dataset utilized in this research is derived from the “Music Emotion Dataset”. It comprises user-generated song lists, along with corresponding label data assigned by users. The dataset encompasses a total of 13,565 song lists, encompassing more than 490,000 individual songs. Each song list is associated with five distinct categories of tags, namely language, style, scene, emotion, and theme.

For the experimentation, a subset of 4,200 music pieces is selected as a sample. Within this sample, 4,927 songs are designated as the feature space, with each song represented as a music vector that characterizes its distribution across the attributes of the song list.

The experimental design entails employing a five-fold cross-validation approach. This methodology involves dividing the dataset into five subsets, and for each iteration, four subsets are used as the training set, while the remaining subset serves as the test set. This process is repeated five times, ensuring that each subset is utilized as the test set once, thereby facilitating robust evaluation and validation of the model’s performance. Cross-validation is valuable when tuning hyperparameters of the model. It allows us to iteratively evaluate the model’s performance with different hyperparameter settings and choose the best combination that generalizes well across various subsets of the data. data distributions may change over time due to various factors (data drift). Cross-validation can help identify how well the model generalizes across different data distributions, providing insights into the model’s robustness.

In the part of user emotion recognition, the listening records of 24,537 users for nearly a week since October 6, 2021 were crawled from “Netease cloud music”, including user ID, music ID and times of listening, which is the corresponding weight.

Evaluation index

F1 is used to measure the accuracy of the model in the multi-classification task, and the value range is 0-1. Before calculating F1, it is necessary to map musical emotion to the emotion class with the maximum corresponding value from the musical emotion distribution in the music-emotion matrix V, and then calculate. The higher the F1 value is, the better the training effect is. In this experiment, each group of tests were carried out three times, and the average value was taken as the results for discussion. The calculation of F1 is expressed as Formula (13) to Formula (15), and the meaning of each variable is shown in Table 1.

(13)${\displaystyle {\text{Precision}}_{\mathrm{mi}}=\frac{{\sum }_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{T}{\mathrm{P}}_{\mathrm{i}}}{{\sum }_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{T}{\mathrm{P}}_{\mathrm{i}}+{\sum }_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{F}{\mathrm{P}}_{\mathrm{i}}}}$

(14)${\displaystyle {\text{Recall}}_{\mathrm{mi}}=\frac{{\sum }_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{T}{\mathrm{P}}_{\mathrm{i}}}{{\sum }_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{T}{\mathrm{P}}_{\mathrm{i}}+{\sum }_{\mathrm{i}=1}^{\mathrm{n}}\mathrm{F}{\mathrm{N}}_{\mathrm{i}}}}$

(15)${\displaystyle \mathrm{F}1=\frac{2\text{*Precision* Recall}}{\text{Precision}+\text{Recal}}.}$

Validity of the algorithms

Figure 3 shows the evolution process of NMF. As can be seen from the figure, the convergence speed of the algorithm without emotional labels is always faster than that of algorithm with emotional labels. The convergence of the two methods is similar, reaching convergence within 400 iterations.

Figure 4 shows the F value of the algorithm on each emotion class. Adding emotional tags had a significant impact on the emotion of “Excitement”, and its F1 increased by 47.62%.“Nostalgia” is generally the best emotional class, while the “quiet” is generally poor, which may be due to the fact that compared with other emotional categories, the performance of “Nostalgia” is generally the best, Listening groups generally have similar cognition on whether music is “Nostalgia”, while “Quiet” and “Cure”, “relaxing” and other emotional categories overlap, which is not easy to distinguish.

In order to further verify the effectiveness of the matrix decomposition method in the music domain data set generated by users, the method in this article is compared with the other five methods: K”-means, support vector machine (SVM), logistic regression (LR), AdaBoost, decision tree (DT). The results are shown in Fig. 5.

The comprehensive analysis of the experimental results reveals intriguing insights into the performance of various models. The non-negative matrix factorization (NMF) technique, when employed with emotional labels, showcases the best overall performance, surpassing AdaBoost, the second-ranking model, by a remarkable margin of approximately 12.5%. This noteworthy difference indicates the superiority of nonnegative matrix factorization (NMF) in capturing underlying patterns and relationships among the emotional features, leading to more accurate classification results.

Notably, k-means exhibits comparable outcomes to logistic regression (LR). The k-means algorithm partitions data into clusters, effectively separating instances with similar features. This behavior aligns with LR’s capacity to find decision boundaries in a linear manner, hence yielding similar results. However, NMF provides an added advantage by decomposing the original matrix into two non-negative matrices that approximately approximate the initial data, thereby mapping features into a hidden space simultaneously. This dual mapping contributes to enhanced classification efficacy.

On the other hand, the decision tree model’s performance falls short, displaying inferior results compared to other models. Decision trees are prone to overfitting and may struggle to generalize well on complex datasets. Consequently, this negatively impacts the model’s recognition effectiveness.

Overall, while some models perform similarly, it is evident that NMF with emotional labels significantly outperforms the rest. Its ability to extract meaningful representations from the data, combined with a dual mapping approach, highlights its potential for various applications, such as sentiment analysis, emotion recognition in text or speech, and user behavior prediction. Further research could explore parameter tuning and feature engineering to optimize the models further, ultimately advancing the field of emotion-based machine learning and data analysis.

User emotion recognition

Music classification

In order to better identify users’ emotions, it is necessary to classify music accurately. The algorithm in Fig. 4 performs poorly in the two clesses of “Relaxation” and “Quiet”. Therefore, the final emotion classification of music that should belong to these two classes is calculated, and the results are shown in Fig. 6.

More than half of the music that should have belonged to the class of “Relaxation” was wrongly assigned to the category of “Cure” and “Nostalgia”, while less was wrongly assigned to the category of “Lonely”. This phenomenon is consistent with cognition. The two class of “Relaxation” and “healing” do overlap. The concert of “Cure” brings people the feeling of “Relaxation”. Similarly, relaxing music can also make people feel “Cure”. Most (40.67%) of the music that should have belonged to the class “Quiet” were wrongly assigned to the class “Cure”, followed by the class “Nostalgia”. Combined with the practical experience in daily life, most of the music under the “Quiet” is pure music, or relatively soft and soothing music, and these music may also have the function of “Cure”, which can regulate people’s mood and emotion and make people calm down. Therefore, this phenomenon is reasonable and conforms to common sense cognition.

To sum up, due to the limitation of data, the reason for this phenomenon is that the limitation of F1. Music contains only one emotion tag, that is, the emotion class with the highest value, while the emotion of music does not necessarily belong to a single class. The above results are consistent with the actual situation, so the reliability of the algorithm in this article is verified.

Analysis of user emotion recognition

When identifying user emotions, the number of nearest neighbor music k of unknown music will affect the result of music emotion recognition, and then affect the result of user emotion recognition. Therefore, parameter experiments need to be carried out around k. Figure 7 shows the trend of F1 with parameter k when identifying user emotions.

The analysis of Fig. 7 reveals a significant trend concerning the performance of F1 with varying values of k. As k increases, F1 demonstrates a consistent improvement until reaching its peak at k = 27, with a remarkable score of 0.509. This suggests that the parameter k profoundly impacts the overall effectiveness of the system. Among the 119 users, a substantial portion of 59 users achieved accurate emotion recognition, resulting in an impressive accuracy rate of 52.7%. This finding underscores the system’s reliability in detecting emotions, positively impacting user experiences and applications in various fields such as sentiment analysis, user feedback processing, and emotional well-being assessments. Further studies could explore optimal k-values and refine the system for even better performance. The result of user emotion recognition is shown in Fig. 8.

The algorithm performs well in the four classes of “Romance”, “Nostalgia”, “Excitement” and “Lonely”. The algorithm performs best in the class of “Lonely”, with the optimal F value up to 0.6415, the highest F value in the category of “Nostalgia” is 0.6071, and the highest F value in the category of “Excitement” is 0.5641. In the “Romantic” category, F value is 0.4286; The highest F value in the “Cured” category was 0.2069, the highest F value in the “Relaxed” category was 0.125 and the highest F value in the “Quiet” category was 0.25. Similar to the previous content, it shows that this is due to the similarity of the three classes, high degree of coincidence and close emotion. For example, “Cure” and “Quiet” music can also make people feel relaxed. Most relaxing music is gentle and soothing, so it is difficult to distinguish them. The other four categories have relatively obvious characteristics, such as “Excitement” music usually has a faster rhythm, while “Lonely” music usually have slow rhythm and sad words. In short, the overall performance of the algorithm in this article is good, which can effectively distinguish emotions under seven emotion categories and identify users’ emotions.