Machine learning and natural language processing to assess the emotional impact of influencers’ mental health content on Instagram

Selection of Instagram posts

The search for Instagram posts to be selected for our research study followed these premises:

• a) Posts had to be made directly by the author and related to the disclosure of mental health issues.

• b) Posts had to be made by Spanish celebrities and influencers and written in Spanish. The selection of these celebrities was based on their ability to have a significant impact, so it was stipulated that they should have more than 100,000 followers on their Instagram profiles.

To conduct this search, profiles of celebrities and influencers on Instagram were analyzed, and relevant news on these topics was reviewed in the press, television, and social media. The search ran from September 2022 to December 2022.

It is noteworthy that, following the search and identification process of posts, the majority of them were made by women. Therefore, the decision was made to exclude the male gender to prevent potential gender biases that could impact AI algorithms. However, in the future, it is planned to integrate the male perspective. Furthermore, another important challenge is the effectiveness of public participation strategies, especially in social networks, due to their inherent biases and the demographic diversity of users. Some groups may not actively participate in social networks, which could lead to biases in the collected data. This is an inherent risk of the research study, as it is primarily aimed at addressing mental health in virtual spaces.

Following these criteria, 20 posts made by women were identified, ultimately selecting the top 10 posts with the most comments. These posts and their associated comments were retrieved using the software tool (Google, 2024). Furthermore, all selected posts exhibited the following characteristics:

• a) The format was consistent, featuring either a photo (five posts) or a video (five posts) accompanied by relevant text.

• b) The content was either related to the manifestation of symptoms of a mental health problem or emphasized the importance of specialized care in such cases.

• c) The most prevalent mental health issues addressed were anxiety and depression.

On the other hand, the ethical aspects of artificial intelligence in the social sciences is a topic of growing interest (Mangino, Smith & Finch, 2021), and the detection of disorders based on comments in social networks poses certain ethical problems in relation to privacy and good practices (Fleming, 2021). However, the objective of the present study is to find out what the reaction to the revelation of a disorder by celebrities is like. In other words, we are trying to find out what the social response to mental health in networks is like, not to make a diagnosis or detect psychopathology. This may allow us to promote in the future a more appropriate use of social networks in relation to mental health, as well as to raise awareness in society about the importance of mental health and its appropriate treatment.

In addition, this study has been carried out taking into account the recommendations of various authors for working with artificial intelligence in social sciences, such as making the results interpretable or being transparent in the development of the algorithms and database (Mangino, Smith & Finch, 2021). Additionally, regarding ethical considerations and data privacy, the study has been approved by the ethics and deontology committee of the University of Valladolid (PI 23-3365). Thus, only public posts and comments were accessed and never private accounts or conversations and, to protect privacy, we anonymised all Instagram comments, filtering out all @mentions and URLs. Table S1 presents a description of the selected Instagram posts, including the number of responses associated with each post, the names of the celebrities/influencers, and their respective number of followers. A total of 21,151 comments were gathered, and the process of categorizing, processing, and incorporating them into a mental health corpus will be describe in next sections.

Description of dataset labelling in emotions

Following previous literature on manual text labelling (Alvarez-Mon et al., 2021; Bograd, Chen & Kavuluru, 2022; Budenz et al., 2020; Delanys et al., 2022) we developed guidelines on the different categories of labelling, emotions in our specific case (GitHub, 2024). Classification began based on the model of primary emotions: fear, anger, sadness, disgust, happiness and surprise (Ekman, 2004). However, this was insufficient to classify the positive emotions reflected in many comments. This is consistent with previous studies, which point out how basic emotions can be oversimplified representations of complex phenomena and lead to lower inter-annotator agreement when categorising in the context of emotive language analyses (Williams et al., 2019). Thus, the categories of positive emotions were expanded based on the model of Plutchik (2001) and the approaches of Fredrickson (2013). Finally, the categories proposed were love/admiration, gratitude, comprehension/empathy/identification, sadness, anger. A deep explanation of each emotions reads as follows:

• Love and admiration: Emotions present in Fredrickson (2013) and Plutchik (2001) models where admiration, approval and love are closely related. They are usually messages with positive content with praise, affection and encouragement; e.g., “you are a champion”, “we love you”, “you come first, we are sure you can do it”.

• Gratitude: Present in recent models (Ekman, 2004; Fredrickson, 2013), the messages imply a sincere appreciation for sharing mental health-related content on social networks; e.g., “thank you for making this problem visible”.

• Comprehension/empathy/identification: Present in Plutchik (2001). They involve interest in and understanding of the message, including self-identification with the situation or context, putting oneself in the other person’s shoes; e.g., “It happens to all of us and we’ve all been through times like this”. They often provoke revelations of mental health problems by claiming to have gone through the same thing. “I understand you so much… I want to get out of this depression”.

• Sadness: This primary emotion (Ekman, 2004) is produced by events that are not pleasant and that denote heaviness. It includes many manifestations of pity for the person; e.g., “poor thing” “what a pity”. There are also comments that include grief at the influencer’s dismissal after her announcement of quitting social media; e.g., “You wil be missed”.

• Anger/contempt/mockery: Initially, the classification of anger/angry and contempt/mockery following the primary emotions model (Ekman, 2004) was proposed. However, after a first classification, we decided to merge the categories, as anger was also present in cases of contempt/mockery. This category involves responses of irritation and attacks on the person as ridiculous and superficial; e.g., “uploading pictures of yourself crying is a higher level of ridiculousness”; “every time you cry you upload a picture… I’m so mad…”; “you’re so disgusting”; “you’re so disgusting”; “you’re so ridiculous…”; “you’re so ridiculous”; “you’re so ridiculous”.

Dataset labelling methodology

The labelling procedure was divided into three phases: an initial phase with a pilot corpus (N = 787 comments), a subsequent phase with all comments on all posts (N = 21,151) and a final phase with the final corpus (N = 2,287) to be used by the machine learning algorithms, as it can be observed in Fig. S1.

Thus, the same methodology and steps were followed in the first two phases, as detailed below.

• Data cleaning: To ensure the relevance of the sample, this process removed comments that contained only acronyms, were made in other languages (not Spanish) or lacked linguistic coherence. In addition, comments naming other people have also been removed, except when the author of the post was tagged and included significant information.

• Removal of emoticons: To analyse the linguistic impact only.

• Expert labelling: The corpus was labelled by two experts blindly (psychologists, trained people) and a third expert reviewed the discrepancies. In case the third reviewer did not break the tie, these comments were discarded from the corpus. In case the third reviewer did not break the tie, these comments were discarded from the corpus. To carry out this process, a set of guidelines on labelling of the different categories was elaborated to train the experts. This guide included descriptions of the categories (emotions) explained in the previous section. In addition, certain labelling strategies were taken into account. For example, one strategy was to identify expressions such as “Thank you”, which often indicated messages of gratitude, although this needed to be checked, as this was not always the case when accompanied by other expressions denoting different emotions. Similarly, expressions such as “I love you very much” often indicated messages of love. Thus, it was also observed that when someone said “how sad”, it probably meant sadness. Finally, swear words or insults reflected comments of hatred or anger.

In the pilot corpus, a discrepancy of only 7.11% (N = 56) was obtained, indicating a strong inter-rate reliability among the experts who participated in the categorization process (Hallgren, 2012). The positive outcomes obtained from the pilot corpus serve as a robust foundation for reproducing the results on a more extensive sample, thereby guaranteeing the consistency of the conducted labelling process. Furthermore, these discrepancies predominantly stemmed from the categories of anger/angry and disgust/mockery. The decision was made to amalgamate both categories due to the frequent co-occurrence of anger in cases involving disgust/mockery/contempt. Some of the messages in which this happened were the following: “All the same, it must be in fashion, haha…” “Taking pictures of yourselves crying to show others is like really unbalanced, seriously, doesn’t the blood flow to your brain properly? Everyone goes through ups and downs, and there are genuine big problems and dramas in everyone’s life”.

As for the second phase, the complete corpus (N = 21,151), the percentage of discrepancy was also around 9.19% (1,943). In this case, the greatest disparity was observed between comprehension/empathy/identification and love/admiration, because some messages included both emotions: “You’re strong, beautiful; never give up. I understand you perfectly, I went through the same thing too. May this crap of life never defeat us. I adore you”. Similar overlaps were observed with other emotions, like love/admiration and sadness, as exemplified in the following message: “My heart breaks at the thought of you being sad. Lots of encouragement, those of us who love and admire you are waiting for you with open arms, much strength, and so much, so much love.” Additionally, instances of overlap occurred between gratitude and comprehension/empathy/identification, as seen in the message: “Thank you very much for sharing it; all of us going through this understand you”. In cases where the third reviewer could not make a decision because there is more than one mixed emotion, these messages were discarded. There were also issues with messages that provided advice like “you should disconnect from social media and rest,” which were ultimately categorized as Neutral, as well as messages referencing a religious theme like “may God be with you”.

In the third phase, corresponding to the final corpus, a meticulous curation process was implemented to form a representative corpus suitable for integration into our machine learning algorithms. Addressing the crucial aspect of class balance in data corpora, the reduction of the large corpus from 15,213 comments to 2,287 comments was executed through a systematic procedure. This reduction aimed to mitigate potential biases arising from imbalances in class distribution, as such imbalances can adversely affect the performance and predictive capacity of classification models. In fact, unbalanced classes cause a problem in predictive models as they tend to focus their attention on the majority class cases and do not correctly classify the remaining classes.

Thus, this process consisted of the next steps:

• Removal of duplicate comments: comments with repetitive content were removed, allowing a single message representative of each recurring topic to be retained. For example, messages containing phrases such as “beautiful, we love you” were condensed into a single message.

• Distribution of comments across categories: This distribution sought to achieve a more balanced representation of comments across all categories, to try to ensure that autistic learning models performed well for all classes. Although there are more comments from some classes than others, the distribution will be much more balanced across the whole corpus.

• Random selection of comments: comments were randomly selected from all posts based on the percentage distribution of each class in the final corpus.

Dataset statistics

Figure S2 shows the descriptive statistics by percentage for each emotion represented in the final corpus. The most predominant emotion in the dataset is Comprehension/Empathy/Identification (28.8%), closely followed by Love/Admiration (28%), both positive emotions. In contrast, the minority emotions in the corpus are Gratitude (9.9%), Sadness (5.4%) and the Neutral class (7.6%). There is a certain imbalance between some categories in the corpus.

Random forest

Random Forest (RF) algorithms are machine learning models based on constructing multiple decision trees and combining their results to make more robust and accurate predictions. Each tree in the forest is trained independently with random subsets of data and features (Breiman, 2001). The advantages include its ability to handle large and complex datasets, reduce the risk of overfitting, provide good generalization and be effective for classification problems (Probst, Wright & Boulesteix, 2018).

A hybrid deep learning approach

A hybrid deep learning model is proposed, leveraging the potential of combining convolutional and recurrent layers. The convolutional layer performs tasks to streamline processing in the recurrent layer, thereby enhancing the overall efficiency of the model. Specifically, the proposed model consists of the combination of the following layers (Fig. S3):

• Embedding layer: It transforms Instagram comments, into vectors to be used by neural networks. In our case, we perform this process by means of word embedding, where each word is represented as a vector, applying the learning word embeddings technique using the corpus itself, to achieve more optimal results. Thus, the embedding layer can be viewed as a dictionary that maps integers, representing words, to dense vectors. When an embedding layer is initialised, these word vectors have random values, but during the training process, these vectors are gradually adjusted through the backpropagation technique. Once training is complete, the embedding space will reflect a structure adapted to the problem being addressed. In addition, to select the final vocabulary for word embedding, the problem corpus is used using the Information Gain (IG) method to preserve the terms with the highest occurrence. IG is preferable to absolute frequency as it measures the frequency of a word within a corpus category relative to its frequency in other categories. Absolute frequency, on the other hand, measures the overall frequency without distinguishing between classes, which does not provide relevant information (Larose & Larose, 2014; Witten, Frank & Hall, 2011).

• Convolution layer (one dimension): It applies filters to capture features and patterns on the input data. Its main function is to detect, learn and extract relevant patterns from the input data by convolution operations, thus eliminating part of the processing of the next layer, the recurrent layer (in our case LSTM). The hyparameters of this layer are the number of filters, and the filter size (kernel size) related to the length of the window and the activation function. Moreover, the rectified linear unit (ReLU) function will be employed.

• MaxPooling layer: This layer decreases the dimensionality of features from previous layers while preserving the most significant information. It operates on a data matrix to produce a smaller feature matrix that keeps the most significant features from the original matrix.

• LSTM layer: It is a recurrent neural network (RRN) that integrates long and short-term memory. The hyperparameters of this layer are number of neurons, dropout rate and recurrent dropout rate. The dropout technique disconnects neurons at each training stage until the optimal value is reached, so these parameters represent this fraction of neurons so that our network will learn just enough and avoid overfitting.

• Dense layer: It provides the model output. The hyperparameters of this layer are the number of neurons and the activation function. The number of neurons is the number of classes to be predicted, in our case six neurons (five emotions, one neutral category). In addition, we select the Softmax activation function, widely used in multi-class classification problems.

As for optimisers and loss functions, the following have been selected:

• Categorical cross-entropy loss function for being largely adopted in multi-classification tasks.

• ADAM optimiser for its versatility, good performance and extensive use in similar problems (Kingma & Ba, 2014).

Bidirectional Encoder Representations from Transformers (BERT) models

The Bidirectional Encoder Representations from Transformers (BERT) models are pre-trained deep learning models based on transformers architecture to learn bidirectional representations of text (Devlin et al., 2018). Using BERT for sentiment analysis involves pre-training the model on a large amount of data and then fitting it to a specific dataset. This allows models to acquire general knowledge of the language and then adjust to better understand the particularities of sentiment analysis in a specific domain, in our case mental health (Wolf et al., 2019). Then we have applied the next BERT classification models:

• RoBERTuito: A linguistic model for social networking texts in Spanish, based on RoBERTa and trained on 500 million tweets (Pérez et al., 2021). It outperforms other models for Spanish such as BETO, BERTin and RoBERTa-BNE (ROBERTuito, 2022).

• XML-RoBERTa (Danevi): A model created to improve multilingual comprehension models through unsupervised training with large amounts of multilingual text. They started from a trained model for the Masked Language Modeling (MLM) task and used a new corpus based on CommonCrawl (Conneau et al., 2019) data in 100 languages for pre-training. As a result, the new model achieved considerable improvements in multi-language classification tasks, up to 23% accuracy with respect to the original mBERT model. The model selected from the Hugging Face repository was trained for emotion classification in text by the user Danevi, so the model will be called by that name (ROBERTa Emotion Twitter, 2021).

Data pre-processing and encoding

Data pre-processing is crucial to ensure the accuracy and efficiency of classification models. Our approach involves then next initial steps in text cleaning and pre-processing:

• Converting to lower case: to provide homogeneity in the text, so that transforming the same word to lower case, such as “Hello” and “hello”, will cause them to be treated the same.

• To remove mentions (@), hashtags (#) and URLs: they are not relevant.

• To remove accents and diacritical vowels: to allow for greater homogenisation vocabulary and avoid including more noise in algorithms.

• To remove punctuation marks: they do not provide any meaningful information.

• To standardise slang/jargon: to transform these expressions into a standardised form of the language to improve data quality. For example, change “d” to “de” (“from” in English), “xa” to “para” (“for” in English), “xq” to “por que” (“because” in English).

• To delete Stopwords: empty words which have no meaning, such as articles, adverbs, prepositions, conjunctions and even some verbs. This elimination allows to focus on the most important information and improve classification performance, as they do not provide meaningful information.

• To apply stemming: a technique to reduce words to their base form. The aim is to simplify and standardise words to improve performance in text classification. For example, the Spanish words “escribir” (“write” in English) and “escritor” (“writer” in English) will be reduced to “escri”. In this search, we will employ the NLTK’s (Snowballstemmer 2021) in Spanish.

By contrast, BERT models include their own data preprocessing: character repetitions limited to a maximum of three, usernames and hastags are converted into a special token, and emojis are replaced by their textual representation. However, the RoBERTuito algorithm was tested with both data pre-processing, and the performance results were quite similar.

Another process consists of tokenization, process by which our texts (Instagram comments) are converted into inputs to be used by classification algorithms. The token can be thought of as the unit for semantic processing at the computational level. We use for RF and deep Learning models the TweetTokenizer tokenizer (Bird, Klein & Loper, 2009) while BERT models use their own tokenizer. Next, all the texts must be transformed into numbers (feature vectors), as machine learning models work with numbers and not with texts or words. In our case, we had to transform both the tokenised Instagram comments and the labels of our classes (emotions in our case).

To transform the messages into numbers, we employed the word embedding technique. In the case of deep learning, we applied the word embedding described in the previous section. In the case of BERT models, tokenisation generates hidden states that form word embeddings for each word, by multiplying these states by a learned weighting matrix. BERT’s word embeddings are distinguished by their ability to capture context, causing the embedding of a word to fluctuate depending on its contextual usage within a sentence. In the case of labels, the One Hot Encoding technique has been used, which encodes the classes as a matrix, in which a “1” is placed in the column of the class (emotion) to which an Instagram comment belongs and a “0” in the other columns. In our case of five emotions (plus a neutral class), we will have a 6-column matrix. However, in the RF model, since it has only one output, instead of using a column for each variable and assigning binary values (0 or 1), a single global variable (one output) is used, corresponding to a specific emotion.

The next step is to define two different datasets: the training set (70% data) and the validation set (30% data), these values being typical in other classification models (Keras, 2023). On the other hand, to find the best ensembles to train models we apply the k-fold cross-validation technique, which consists of performing k iterations and the model is trained and evaluated k times. Finally, we apply the EarlyStopping technique to regulate overfitting and prevent models from losing generalisation (Keras, 2023), so that the training of a model stops when the training loss metric stops improving. The classification models have been implemented in Python 3.1 using Keras (2023) and TensorFlow (2023). libraries (Abadi et al., 2016). All the models were run on the Google Colab Platform, which allows running Jupyter notebooks on shared hardware provided by Google.

Results and comparison of the classification models: RF, deep learning, BERT

This section describes the accuracy results of the classification models. In addition, we will consider other metrics to assess the performance of the models in each class separately, such as Recall, Precision and F1-score, which are especially important when the dataset is unbalanced. For this purpose, the best values of the hyperparameters of each model that optimise all the quality metrics will be sought.

Limitations

In terms of potential limitations for the development of the study, it is worth highlighting the speed and temporality that are associated with social networks, since high-impact posts are often deleted or remain accessible for a short time, and social networks trends are particularly fast and unstable. To reduce this risk, a specific period has been selected, and posts of interest will be actively tracked throughout the study. Although showing a snapshot of a time is a limitation, the study is designed so that new posts can be integrated to feed back into the algorithms and improve or extend their effectiveness. In addition, another important challenge is the effectiveness of public participation strategies, especially in social networks, due to their inherent biases and the demographic diversity of users. Some groups may not actively participate in social networks, which could lead to biases in the collected data. This is an inherent risk of the research study, as it is primarily aimed at addressing mental health in virtual spaces.

Another potential limitation is the subjectivity of emotions and the lack of a universal classification, which may bias the prediction made by AI algorithms, but different classifications of emotions have been studied, using categorizations based on scientific findings and well-established in the literature that best fit our study. Another limiting issue is related to the balance between classes in labelled corpora, which can be prejudicial to the performance of classification algorithms. The presence of unbalanced classes poses a problem in predictive models as they tend to focus their attention on the majority class cases and do not correctly classify the rest of the classes. To address this problem, a careful selection process of the comments in the corpus has been carried out to try to mitigate this type of factor.

An additionally risk to be addressed is the bias of the corpus labellers, which is minimised as they have been trained in advance and the process was carried out by a set of two independent experts and reviewed by a third party (peer review process). The last limitation of the research concerns the language of the comments, as only messages in Spanish responding to posts in this language were processed, so the context of the study is Spanish. Future analyses in other contexts and cultures can be analysed to compare cultural and language-dependent differences.