Sentiment classification for employees reviews using regression vector- stochastic gradient descent classifier (RV-SGDC)

TextBlob

TextBlob is a well-known python library that provides sentiment orientation of the text under analysis (Chaudhri, Saranya & Dubey, 2021a). It assigns a P_score which is a float value within the range, R_(Pscore) = [ + 1.0,  − 1.0], in which P_score =  + 1.0 refers to positive polarity and P_score =  − 1.0 refers to negative polarity. In addition to this, TextBlob also provides a subjectivity score which quantifies the text under analysis as factual information or a personal opinion (Chaudhri, Saranya & Dubey, 2021b). This analysis is only concerned with the polarity score thus we did not integrate the subjectivity score. The threshold for TextBlob was set to 0, thus categorizing reviews with Overall P_score > 0 as positive reviews (satisfied) and Overall P_score < 0 as negative reviews (unsatisfied). Sample of polarity score assigned by TextBlob to employees’ reviews is shown in Table 4.

Term Frequency-Inverse Document Frequency (TF-IDF)

TF-IDF is an information retrieval algorithm that statistically measures the relevance of a term in a document (Jiang et al., 2021). It works by weighing a term t’s frequency (tf) and its inverse document frequency (idf). Term frequency refers to the number of occurrences of a term in a document and inverse document frequency refers to the significance of that term in the whole dataset (D). In this study, we used the scikit-learn library, TfidfVectorizer, in which TF-IDF assigns weight to each term t in correspondence to its significance in the review R under analysis in the following manner: (1)${\displaystyle tfidf\left(t,R,D\right)=tf\left(t,R\right)\times idf\left(t,D\right)}$

where, (2)${\displaystyle tf\left(t,R\right)=log\left(1+freq\left(t,R\right)\right)}$ (3)${\displaystyle idf\left(t,D\right)=\frac{logN}{\left(df+1\right)}.}$

Bag of Words (BoW)

The BoW is a well-known feature extraction technique that represents text data disregarding word order and grammatical details (Sahin, 2021). It is widely used for NLP tasks, text classification, topic modeling, and information retrieval. This approach is easy to implement. Each instance in this feature extraction technique is tokenized to construct a vocabulary then, frequency of each token in the vocabulary is calculated (Boag et al., 2021). In simple words, BoW converts text of variable lengths into a vector of fixed length thus making it easy for ML models to work with data. For the implementation of BoW in our experiments, we integrated CountVectorizer which is a sci-kit-learn library in python.

Global Vector Representation of Words (GloVe)

GloVe was released by a group of NLP researchers from Stanford for vector representation of words in continuous space (Li, He & Chen, 2021). It maps words into relevant space by the distance between words and their semantic similarity. The GloVe is widely used for entity name recognition, machine translation, and many other NLP tasks. It mainly works by constructing a co-occurrence matrix M, in which M_ij represents the frequency of word i appearing in some context of word j, in which co-occurrence is calculated by moving a context size window over each sentence in the text. The study import GloVe embedding from ‘zeugma’ library to employ in our experiments.

Logistic regression

LR is a statistical ML model which models a set of input features (X: input) into target variables (Y: output) by means of a sigmoid function which is an ‘S’ shaped curve and restricts Y in the range of 0 and 1 (De Cock et al., 2021). Sigmoid function σ:R⟶(0, 1) is defined as: (4)${\displaystyle \sigma \left(X\right)=\frac{1}{1+e^{\left(-X\right)}}}$

where e is the base of the natural log. LR with its easy implementation produces efficient results in the case of binary classification (Rupapara et al., 2021). In our experiments, we have integrated various parameters in LR such as multi_class, random_state, and C. Multi_class is set to ovr which is the best choice for binary classification, random_state is set to 100, and C is set to 3 for strong regularization.

Random forest

RF is a meta estimator which utilizes the bagging method for learning patterns from data. It is an ensemble of decision trees that are built on numerous sub-samples of the dataset. It produces optimized prediction results by pooling outputs obtained from each decision tree in the ensemble (Chen et al., 2021). It controls over-fitting by adding additional randomness to the model, it does so by considering the most appropriate features from a random subset of features while splitting a node instead of utilizing the most important feature. One of the most significant tasks in the construction of a decision tree in RF is the selection of a significant attribute as a root node. For this purpose, two techniques including Gini impurity and Information Gain are used Baranauskas, Oshiro & Perez (2012). The current study involves Gini impurity as the criterion to quantify the split of a node. Gini impurity for a node can be calculated as: (5)${\displaystyle \sum _{x=}^N{f}_x\left(1-f_x\right)}$

where, f_x is the number of occurrences of target attribute x at a node and N is the number of unique attributes. Several hyper-parameters are utilized for tuning of RF to acquire the best results. In our experiments, we set n_estimators, which defines the number of decision trees to be built in the ensemble, to 100. The randomness of samples bootstrapped in the construction of decision trees in the forest is set to 50, and the depth of each decision tree is set to 250 as shown in Table 5.

AdaBoost classifier

AC, also referred to as Adaptive Boosting, is an ensemble method that reassigns weights to each classified instance of the learner and higher weights are assigned to instances that had been classified incorrectly (Wu et al., 2020). AC constructs an ensemble of learners in a sequential manner in which each successive learner is constructed from preceding learners which results in less variance and bias of the models. It employs boosting for transforming a weak learner into a strong learner by concentrating on instances that are wrongly classified by a preceding weak learner. For a given dataset {(x₁, y₁), …, (x_n, y_n)} where each instance x_i has a corresponding target variable y_iɛ{ − 1,  + 1}, weak learners {k₁, k₂, …, k_m} are combined in the form: (6)${\displaystyle C_{\left(t-1\right)}\left(x_n\right)=w_1{k}_1\left(x_n\right)+,,,+w_{\left(t-1\right)}{k}_{\left(t-1\right)}\left(x_n\right)}$

where, w₁, , , w_(t−1) is the weight assigned to each predicted instance and C_(t−1)(x_n) represents the target variable at (t − 1)th iteration which is further extended to construct a strong learner by adding another weak learner k_t with another weight w_t. (7)${\displaystyle C_t\left(x_n\right)=C_{\left(t-1\right)}\left(x_n\right)+w_t{C}_t\left(x_n\right).}$

The hyper-parameter setting for AC used in our experiments is shown in Table 5. We set n_estimators to 300 and random_state to 50 to acquire optimized results.

Multilayer perceptron

MLP classifies an input vector ′x′ by multiplying it with a weight ’w’ and adding bias ’b’ to it, such as, output:w ×x + b. MLP is restricted to the linear mapping of input and output variables, whereas, perceptron in MLP is capable of carrying out the classification of data which is not linearly separable (Car et al., 2020). MLP is an extended feed-forward neural network that maps a non-linear relationship (f:R^m⟶Rⁿ) between the input layer of m dimensions and output vectors of n dimensions. Additionally, MLP consists of an arbitrary number of hidden layers in which neurons are trained by integrating the back-propagation technique. Computationally, a set of {n_i|n₁, n₂, …, n_m} input features from input layer are assigned a weight in the hidden layer such as: w₁n₁ + w₂n₂ + , , ,  + w_mn_m in relation to a non-linear function (f: R → R). Current study utilizes rectified linear unit function (relu: f(x) =max(0,x)) as an activation function. The hyper-parameter setting for MLP in our experiments is shown in Table 5.

Extra tree classifier

ETC is an ensemble of unpruned decision trees that involves an arbitrary subset of features for splitting of nodes (Bhati & Rai, 2020). Unlike RF, it integrates whole data in the construction of a decision tree instead of bootstrapping data. It involves two main parameters such as the number of randomized input features selected at each node(K), the minimum size of sample required for splitting a node n_min, and the number of decision trees in the ensemble (M). Decision trees in ETC are less likely to be correlated due to the randomized selection of points of the split. ETC averages predictions of decision trees in the ensemble to produce a final prediction in case of regression (Alsariera et al., 2020). This study concerns binary classification thus pertaining to majority voting of decision trees’ predictions to output a final prediction. There are several hyper-parameters involved in the tuning of ETC. In our experiments we have set n_estimators which states the number of trees in the ensemble to 200, random_state to 50, max_depth of each decision tree to 150, and Gini impurity as the criteria for node split.

Support vector classifier

SVC is a pattern-based classifier which utilizes linear kernel function to map input vectors x_i into a high dimensional vector space (Rezaeinia et al., 2019). It then creates a linear hyper-plane with the optimized margin between the target classes y_i. The linear kernel function integrates the following mathematical computation for pattern recognition. (8)${\displaystyle K\left(x_i,y_i\right)=x_i^{\prime }{y}_i.}$

Hyper-parameters concerning SVC in this study are shown in Table 5. Regularization parameter C is set to 1.0 for strong regularization and random_state, to shuffle the data for estimating probability, is set to 500 for optimized results.

Stochastic gradient descent

Gradient descent (GD) is a well-known optimization technique that learns the optimized values of models’ parameters at each iteration to minimize the cost function (c^f) (Rustam et al., 2019). SGD is a variant of GD which concerns itself with random probability (stochastic) such that, at each iteration, a single sample is selected for the training of the model (Tan et al., 2021). It requires significantly less training time for finding c^f of only one training sample xⁱ at each iteration to reach local minima. It does so by updating the parameters of the model for each xⁱ and corresponding target class yⁱ. (9)${\displaystyle {\theta }_j={\theta }_j-\alpha \left(y^{i^{{}^{\prime }}}-y^i\right)x_j^i}$

where θ_j is the parameter and α is the learning rate of the model. SGD employs several hyper-parameters which supports its performance on data under analysis. In this study, loss function hinge was selected as default with l2 regularization. Maximum epochs were set to 1000 and the criterion of stopping iteration was set to 1e − 3 as shown in Table 5.

Regression Vector Stochastic Gradient Descent Classifier (RV-SGDC)

This study proposed an RV-SGDC model which is a combination of three individual models LR, SVC, and SGDC. The RV-SGDC outperforms all other models because of its ensemble architecture. We combined these models using majority voting criteria which mean that individual models will give their predictions for the target class and then the most predicted class by the models will be considered as the final target class. We combined LR, SVC, and SGDC because of their better performance individually as compare to other models and these models produce a more efficient and accurate model with the hard voting combination. We can describe RV-SGDC mathematically as: (10)${\displaystyle L{R}_{prediction}=L{R}_{trained}\left(Data\right)}$ (11)${\displaystyle SV{C}_{prediction}=SV{C}_{trained}\left(Data\right)}$ (12)${\displaystyle SGD{C}_{prediction}=SGD{C}_{trained}\left(Data\right)}$

And (13)${\displaystyle RV-SGD{C}_{prediction}=model\left\{L{R}_{prediction},SV{C}_{prediction},SGD{C}_{prediction}\right\}.}$

Here LR_prediction, SVC_prediction, SGDC_prediction are the predictions by the individual models and RV − SGDC_prediction is final prediction using majority voting between individual models’ predictions. The architecture of proposed model is shown in Fig. 1.