A novel text sentiment analysis system using improved depthwise separable convolution neural networks

Typical text sentiment analysis methods

The fundamental component of text sentiment analysis is the classification of sentiments according to the information that is expressed by text data. Standard approaches to sentiment analysis can be broken down into three distinct categories: those that are founded on sentiment dictionaries, those that are founded on machine learning, and those that are founded on deep learning. The use of sentiment dictionary method for sentiment classification is the most primitive technical means in text sentiment classification. The advantage of this method is its flexibility, because we can build different sentiment dictionaries according to different scenarios and different needs. However, as a result of advances in technology and shifts in social mores, the text is becoming progressively less structured, and the emotional underpinnings of the majority of texts are no longer readily apparent. The use of a sentiment dictionary for the purpose of sentiment classification has not produced the effect that users had anticipated; hence, at this time, the method of sentiment classification that is based on machine learning was developed and is now extensively utilized. The text sentiment analysis shown in Fig. 1 is presented in the form of a schematic diagram based on sentiment dictionary.

When compared to the method that is based on a sentiment dictionary, the method that is based on machine learning has a significant improvement in the effect of text sentiment classification; however, both methods show their respective advantages in a variety of tasks that involve sentiment classification. However, each of these approaches do have some potential downsides. The update speed of the method of a sentiment dictionary is slow and cannot keep up with the times, which means that the accuracy of sentiment classification is closely related to the size of a sentiment dictionary and the accuracy of manual annotation. In addition, the size of a sentiment dictionary directly influences the accuracy of manual annotation. There is a significant amount of human processing effort involved in the feature engineering of sentiment categorization systems that are based on machine learning. When there is an excessive volume of data, the efficiency will be very poor or it may even be impossible to achieve. Despite the fact that the machine learning model is straightforward, it has a limited capacity for generalization and a weak classification effect. The text sentiment analysis based on machine learning is shown here by a schematic, which can be found in Fig. 2.

Deep learning algorithms outperform sentiment dictionaries and machine learning algorithms in classification performance. However, deep learning algorithms also have their inherent problems, such as time-consuming algorithm operation, high algorithm complexity, too many parameters to be adjusted, and algorithm sensitivity to parameters. Scholars have also successively optimized and improved deep learning algorithms from the above levels to further improve their sentiment analysis effects on texts. Figure 3 presents the results of a sentiment analysis of text based on deep learning.

Convolutional neural network model

The CNN is an example of a feedforward neural network, which is a type of neural network that is highly representative of deep learning. Compared with other neural networks, the convolution and pooling structures of CNN require relatively few parameters to debug, and have excellent performance in sentence matching recommendation systems, semantic processing, text classification and other fields. In 2014, Kim (2014) proposed the Text-CNN model, which uses multiple convolution kernels to extract features similar to N-gram for text, and achieved good results in the field of short text classification. The overall network architecture of the model is shown in Fig. 4.

The Text-CNN model is made up of four distinct components, which are the input layer, the convolution layer, the pooling layer, and the fully connected layer. The matrix that corresponds to the document serves as the input for the model at the model input layer. Generally, the word is first mapped into a word vector with a fixed length of d through the Word2Vec model, and then the document is truncated or supplemented into a text with a length of l. In this way, the entire document can be represented in the form of a matrix of size l * d. The contextual N-gram features of sentences are then captured by convolutional layers. The difference from the commonly used image convolution kernels is that the Text-CNN model transforms the traditional k × k convolution kernels into k × d convolution kernels suitable for text classification, where k is the convolution kernel window size. This can ensure that the word vector information will not be truncated during the convolution process and maintain the integrity of the semantics. The feature map that is produced by the convolutional layer is then sent to the pooling layer, where the downsampling operation is used to select features and filter information in order to accomplish dimensionality reduction, the removal of redundant information, and the simplification of network complexity.

Text sentiment analysis model based on improved depthwise separable convolution neural networks

Improved depthwise separable convolution neural networks

A word vector is a comprehensive reflection of the meaning of a word. Unlabeled data training is destined to be a semantic representation that combines polysemy and all contextual information. So even the simplest linear model, such as Fast-Text. The model can also achieve good classification results. In view of the problem that the existing deep learning model cannot make full use of the word vector features and the model complexity is too high, this research divides the traditional convolutional layer into two layers, one layer performs word embedding convolution, extracts word vector features, and the other layer perform regional convolution to extract contextual features.

The depthwise separable convolution algorithm first appeared in the Mobile Net (Kazerouni, Dooly & Toal, 2021) model in the field of image classification. In Qin et al. (2019), a decomposable convolutional network is used to realize the lightweight conversion of the traditional SSD network, which greatly reduces the complexity of the network. Figure 5 presents the depthwise decomposable convolution structure in the Mobile Net model. The model employs depthwise convolution filters and 1 × 1 convolution filters instead of traditional image standard convolution filters. But the image convolutional network structure is not suitable for text. Since for an image, the correlation between any pixel and surrounding pixels is probabilistically equal, it is feasible to use a convolution kernel of size C_w × C_w. However, for text, the word vector of a word and the context in which the word is located are completely different considerations, and the two cannot be confused. Therefore, it is inappropriate to use a convolution kernel of size C_w × C_w or C_w × d, where C_w represents the size of the convolution kernel, and c represents the dimension of the word vector.

Aiming at the structural features of text and the idea of depthwise separable convolution, this study proposes a separable convolution algorithm suitable for text. First, the text separable convolutional network retains the advantage of low complexity of the image separable convolutional network. Second, the model realizes the hierarchical extraction of text features, which ensures the integrity of the word vector information during the convolution operation. It can make full use of text features and reduce the semantic ambiguity caused by polysemy and other phenomena. Figure 6 shows how the standard text convolution filters are decomposed into word embedding convolution filters and region convolution filters.

First, after text preprocessing and vectorization operations, the size of the sentence is m ×c × 1. The standard text convolution kernel size is C_w × c × 1 × S, where m is the text length, c is the word vector dimension, and C_w is the window size of the convolution kernel, S is the number of output channels. Therefore, the computational cost of text standard convolution is (1)${\displaystyle m\times c\times 1\times C_w\times c\times 1\times S.}$

The text separable convolution algorithm emphasizes that the connection between the dimensions of the word vector is different from the contextual connection between words. Therefore, first use word embedding convolution to break the interaction between regional convolutions, so that the convolution kernel only extracts a single word vector feature. In this way, the model only pays attention to the word itself in the convolution operation, so as to obtain as much word vector information as possible, so that the model can learn to distinguish the phenomenon of polysemy. Second, regional convolution is applied to obtain regional features of each word to analyze the local content of its context. The word embedding convolutional layer uses a convolution kernel with a size of 1 × c × 1 ×T, and the calculation amount of the convolution process is: (2)${\displaystyle m\times c\times 1\times 1\times c\times 1\times T.}$

After the first iteration of the convolutional algorithm, the size of the feature map that is obtained is 1 × 1 ×T. In the second step, the size of the regional convolution kernel is calculated as follows: C_w × 1 × T × S, where T is the number of word embedding convolution kernels and S is the number of regional convolution kernels. In other words, the size of the regional convolution kernel is determined by the product of these two numbers. Applying this to the feature map above, the computational cost of this convolution process is: (3)${\displaystyle m\times 1\times 1\times T\times C_w\times 1\times T\times S.}$

Therefore, the total computational cost of the text decomposable convolutional network is: (4)${\displaystyle m\times c\times c\times T+m\times T\times C_w\times T\times S.}$

By simplifying Eq. (1), the complexity of the traditional text convolutional network becomes: (5)${\displaystyle m\times c\times C_w\times c\times S.}$

Therefore, by expressing the traditional text convolution as a two-step convolution process of word embedding convolution and region convolution, the reduction in computation is as follows: (6)${\displaystyle \frac{m\times c\times C_w\times c\times S}{m\times c\times c\times T+m\times T\times C_w\times T\times S}=\frac{T}{S\times C_w}+\frac{T}{c\times c}.}$

Taking S = T, and S is the same order of magnitude as c, and C_w is much smaller than c, Eq. (6) is simplified as follows: (7)${\displaystyle \frac{T}{S\times C_w}+\frac{T}{c\times c}\approx \frac{1}{C_w}.}$

Equation (7) shows that the complexity of the text separable convolution algorithm is $\frac{1}{C_w}$ of the traditional convolution algorithm. This shows that the practical application value of the separable convolution network in the field of text analysis and processing is much higher than that of the traditional convolution algorithm from the perspective of mathematical analysis. Therefore, from the perspective of model complexity, the text sentiment analysis model based on separable convolutional network is more superior than other deep learning models.

Text sentiment analysis model

A model for analyzing the sentiment of text is built by combining the detachable convolutional network described in Section 3.1 with Softmax. Figure 7 provides a visual representation of the structure of the text sentiment analysis model.

Five layers make up the entire model: the input layer, a word embedding convolutional layer, a region convolutional layer, a pooling layer, and a fully connected layer. The conventional CNN methodology is used for the processing of the first input layer here. The preprocessed sentence is mapped into a vector matrix through the Word2Vec model, which is convenient for subsequent convolution operations. Let x_i ∈ R^c denote the word in the sentence with the i th vector of dimension c. x ∈ R^m×c denote the input sentence with sentence length m. Therefore, the training text of length m can be expressed as: (8)${\displaystyle x=\left[x_1,x_2,\dots ,x_m\right].}$

The purpose of the second layer of word embedding convolutional layers is to handle word sense and contextual meaning separately. Doing so enables the convolution kernel to focus on analyzing the semantic content of words, thereby extracting fine-grained textual features. This part of the word embedding convolution operation weight Z1 ∈ R^1×c. A convolution kernel is applied to a single word vector to generate new features that contain only information about the single word. As an illustration, a word feature f 1 is produced when a word x_i is used: (9)${\displaystyle f{1}_i=h\left(Z1\times x_i+g1\right)}$

where g1 ∈ R is the bias term, which is mainly used for the word embedding convolution kernel. h refers to nonlinear functions such as tangent, sigmoid, etc., because RELU can avoid gradient explosion and gradient disappearance within a certain range. In the model, RELU is used as the activation function h. Equation (9) is applied to all word embedding convolution kernels to generate feature map f 1: (10)${\displaystyle f1=\left[f{1}_1,f{1}_2,\dots ,f{1}_m\right].}$

f1 ∈ R^m, where f 1 represents the word vector information feature of the text after word embedding volume. The third layer is the regional convolution layer, whose purpose is to obtain the contextual features of words based on the word information of the previous layer. The filter Z2 ∈ R^C_w×1 of this partial region convolution operation is applied to the c1 feature map to capture the contextual features of each word. For example, feature f 2_i is generated from window $\left[f{1}_i,\dots ,f{1}_{i+C_w-1}\right]$ by Eq. (11): (11)${\displaystyle f{2}_i=h\left(Z2\times \left[f{1}_{i:i+W_c-1}\right]+g2\right).}$

g 2 is a bias term for the f 1 feature map. The activation function h adopts the RELU function. The filter is applied to each possible window of the word embedding feature, and the resulting feature map is as follows (12)${\displaystyle f2=\left[f{2}_1,f{2}_2,\dots ,f{2}_{m-W_c+1}\right].}$

Where f2 ∈ R^m−C_w+1, which represents all the contextual information features of the text after regional convolution.

The fourth layer of pooling layer, its purpose is to further extract f2 features and realize feature dimensionality reduction. Improve model generalization ability by reducing model complexity. The most common pooling operations are average pooling and max pooling. In this article, the maximum pooling is selected, that is, the maximum value in the feature map is selected as the final value after pooling in this area, because the maximum pooling can reduce the estimated mean shift caused by the error of the convolutional layer parameters. Through the pooling layer, the most important context information is selected to represent the entire text feature, and several one-dimensional vectors are obtained. $\tilde{f}2$ isused to represent the max pooling feature: (13)${\displaystyle \tilde{f}2=max\left\{c2\right\}.}$

Due to the existence of the number of regional convolution kernels, $\tilde{f}2\in R^N.\tilde{f}2$ characterizes the finally obtained comprehensive text features. The last layer is the fully connected layer. The $\tilde{f}2$ feature applies dropout to the fully connected layer, and the layer weight vector Z 3 is constrained by L2 regularization. Dropout can make the activation probability of some neurons fixed on the e value, and the e value ranges from 0 to 1. Therefore, the model does not depend too much on some local features in the forward transmission process, which makes the model more robust and generalizable. Added L2 regularization to prevent overfitting more effectively. That is, Eq. (15) can be used instead of Eq. (14) to represent the output unit y of the forward transmission.

(14)${\displaystyle y=Z3\times \tilde{f}2+g3}$ (15)${\displaystyle y=Z3\times \left(\tilde{f}2\cdot v\right)+g3}$

where g 3 is also a bias term. ⋅ isthe element-wise multiplication operator. v ∈ R^S isthe masking vector of the Bernoulli random variable. A probability of 1 indicates dropout, realizing that gradients are propagated only through unmasked units. Finally, the output unit y gets the final classification label through the Softmax classifier.

Experimental setup

This article plans to grasp the emotional state of college students through text emotion recognition, so as to better assist colleges and universities to manage college students’ study and life in school. As a result, the text sentiment data of college students was compiled by the authors of this research in order to validate the usefulness of the model that was applied. On the other hand, in order to conduct a more objective analysis of the performance of the model that was utilized, the experimental analysis on public datasets that this article presents is carried out. Whether it be an experiment using a publicly available dataset or one with a dataset that the author has created themselves, the experimental environment described in this study adheres to the environment provided in Table 1:

Public dataset experiments

The public dataset used in this article is Chnsenticorp. The number of sentiment classifications for this dataset is 2, positive and negative. The sentiment label of positive text is 1, and the sentiment label of negative text is 0. Some experimental data samples are shown in Table 2.

A total of 60% of the dataset is used as the training dataset and 40% as the test dataset. The evaluation indicators use Accuracy, Precision, and Recall. Several classic deep learning models are selected for the comparison model, namely CNN (Enamoto, Li & Rocha, 2021), recurrent neural network (RNN (Kim, Jo & Lee, 2019), LSTM (Schmitz et al., 2021), BiLSTM (Noh & Cho, 2021), Text-CNN (Zhang et al., 2021)). Table 3 displays the results of the experiments conducted on each model using the publicly available dataset:

Following are some of the findings that can be drawn from an examination of the data shown in Table 3: (1) for the Accuracy indicator, the results obtained by each model exceeded 0.85. The experimental data obtained by several classical algorithms such as CNN, RNN, LSTM, and BiLSTM were similar, and the BiLSTM model obtained the best results. BiLSTM adds a gate mechanism and memory unit on the basis of RNN, which effectively prevents gradient explosion and gradient disappearance. At the same time, it better captures long-distance dependencies, and BiLSTM can capture bidirectional semantic dependencies. This does, in fact, result in an improvement in the overall performance of the model. Text-CNN has implemented specific enhancements aimed towards the text data. The results of the experiments make it abundantly evident that the suggested model has vastly improved the classification results of text data; specifically, it has improved by 1.93% on the basis of Text-CNN. This can be seen clearly from the results of the experiments. (2) The Precision index values obtained by each model are lower than the Accuracy value. The reduction of RNN is significantly lower than that of CNN, which is why this article chooses CNN as the basic model. For the Text-CNN model that performs well on the Accuracy indicator, its Precision indicator value is obviously not ideal. The model used in this article is improved by 3.08% on the basis of Text-CNN. (3) For the Recall indicator, the proposed model is improved by 8.29%, 6.8%, 8.3%, 7.1%, and 4.21%, respectively, compared with CNN, RNN, LSTM, BiLSTM, and Text-CNN. Good performance shows that the model has good robustness.

Self-made dataset experiment

This study crawled the data in the on-campus post bar of a university in Jiangsu Province, and randomly selected 1,000 posts from the data. A total of four people categorize posts sentimentally into two categories, positive and negative. This post data is kept only if four people give the same category tag, otherwise, the post will be discarded. Finally, 16,268 posts were retained as experimental data. There are 12,000 posts that are chosen at random to be used as training data, while the remaining posts are used as test data. The comparative model agrees with the model that was utilized in the experimental data that is freely available to the public. The results of the experiments performed on the data collected by oneself are presented in Table 4 as follows:

In general, the experimental findings on the self-made data set are superior to the experimental results on the public data set. This is due to the fact that the emotional tendency of the text data selected when making the data is pretty clear. All of the models have a recognition accuracy that is more than 0.9. This demonstrates that deep learning models are capable of achieving very good outcomes in the process of analyzing the sentiment of text. RNN has the worst performance when it comes to the recognition performance of the self-made dataset. The experimental findings achieved by the BiLSTM model are comparable to those obtained by Text-CNN. The suggested model exhibits an increased accuracy of 2.55% and 2.27% when compared to the BiLSTM and Text-CNN models, as well as an improvement in precision of 2.21% and 2.19%, and an increase in recall of 4.5% and 3.95% respectively. All three of the assessment indexes point to the fact that the suggested model has the most successful influence on emotion recognition in written text. This shows that the proposed model performs hierarchical convolution on text features, and can indeed achieve refined extraction of word vector information and context information. Doing so avoids semantic confusion and reduces the complexity of convolutional networks. The improvement of the model can solve the text sentiment analysis task well, which further verifies the effectiveness of this work.