Topic2features: a novel framework to classify noisy and sparse textual data using LDA topic distributions

Tokenization and lemmatization

Tokenization is the process of breaking the document or tweets into words called tokens. A token is an individual part of a sentence having some semantic values. Like Sentence hurricane is coming would be tokenized into ‘hurricane’, ‘is’, ‘coming’. We have utilized the Spacy function with the core English language model for tokenization and lemmatization (https://spacy.io/usage/models). The beauty of this Spacy function is that it gives you part of speech detail of every sentence, and you can choose from that which part of speech you need for further processing in the specific context. Spacy is capable enough to also give sentence dependencies in case you need them while performing graph embeddings. After tokenization, we need to see which part of the sentence we need and also need to extract the words into their original forms. Both the lemmatization process and the stemming process are used for this purpose. Many typical text classification techniques use stemming with the help of port stemmer, and snowball stemmer, such as words ‘compute’, ’computer’, ’computing’, ’computed’ would be reduced into word ‘comput’, a little drawback with stemming is that it reduces the word into its root form without looking into it the word is found in the dictionary of that specific language or not, as you can see ‘comput’ is not a dictionary word. There comes the lemmatization, Spacy; we performed the lemmatization. Lemmatization also reduced the word into its root form but by keeping in mind the dictionary database. With lemmatization the above examples of words (‘compute’,’ computer’,’ computed’,’ computing’) would be reduced to root form as (‘compute’,’ computer’,’ computed’,’ computing’), respectively, by keeping in mind the dictionary. While implementing the lemmatization part, we keep the sentence with words having only the ‘nouns’, ‘adjectives’, and ‘verbs’, which is useful if you need to be more specified about the LDA topics and in this way your topic distributions make more sense.

Bigrams and trigrams

Sometimes in large and sparse texts, we see the nouns or adjectives that make of multiple words, so to make the semantic context of words into sentences we need bigrams and even trigrams so that they will not break into single separate unigram tokens and lost their meaning and semantic of a sentence. Bigrams is an approach to make words that are of two tokens to remain in their semantic shape so that sentence contextual meaning would not be lost. We achieved this through Genism’s phrases class (https://radimrehurek.com/gensim/models/ldamodel.html), which allows us to group semantically related phrases into one token for LDA implementation. Such as ice_cream, new_york listed as single tokens. The output of genism’s phrases bigram mod class is a list of lists where each one represents reviews, documents, or tweets, and strings in each list is a mix of unigrams and bigrams. In the same way, for the sake of uniformity of three phrases words tokens, we applied trigrams through genism’s phrases class to group semantically related phrases into single tokens for LDA implementation. This normally mostly applies to country names such as united_states_of_America, or people’s_republic_of_china, etc. The output of genism’s phrases trigrams is a list of a list where each list represents review, document, or tweets, and strings in the list is a mix of, unigrams, bigrams, and trigrams. To make the LDA model more comprehensive and specific we applied the bigrams and trigrams.

Sparse vector for LDA model

Once you have the list of lists of different bigrams, and trigrams then you pass into Genism’s dictionary class. This will give the representation in the form of a word frequency count of each word in strings in the list. Genism’s LDA implementation needs text as a sparse vector for the LDA model. We have used Genism’s library doc2bow simply counts the occurrences of each word in documents and creates and returns the spare vectors of our text reviews to feed into the LDA model. The sparse vector [(0, 1), (1, 1)] therefore reads: in the document Human–computer interaction, the words computer (id 0) and human (id 1) appear once; the other ten dictionary words appear (implicitly) zero times.

Apply LDA Model

To apply the LDA model, there is a specific representation of the content that we need in the form of corpus and along with the corpus, we need the dictionary that assists that corpus. For different LDA models, we create a different type of corpus, with unigrams, bigrams, and trigrams. LDA model is specifically described in detail in Fig. 1.

LDA Model selection

LDA model selection was the most difficult task, as it can ultimately impact the results of supervised classifiers. Therefore, to choose the best LDA model with many numbers of topics in the model was a time-consuming task. Finding the exact number of topics suited for a better LDA model was the main focus of previous studies (Greene, O’Callaghan & Cunningham, 2014). The first technique was manual, which is to choose the different number of range of topics and check and investigate the results if it makes any sense. The second one was analyzing the coherence score metrics of LDA models more coherence increase means better model. Then we also explored the models by giving several various topics and every topic distributions results and vectors feed into supervised algorithms and check which one gives the better results in terms of F1 score. Approach one was very time consuming, the second one was to see the coherence score but that just check the topic identifications have not a large impact on supervised algorithms results, the third approach seems suitable in our context but our main purpose was to classify the documents/reviews with best results. Genism also provides a Hierarchical Dirichlet process (HDP) (https://radimrehurek.com/gensim/models/hdpmodel.html) class that used to seek the correct number of topics for a different type of datasets it is not necessary to type the number of topics in HDP class and automatically seeks the number of topics based on data. It is only necessary to run this for a few times, and if it provides the same results with the same number of topics again and again then those number of topics are perfect learning topics for your type of data.

Hierarchical Dirichlet process

According to Genism’s documents, the hierarchical Dirichlet process (HDP) is based on stick-breaking construction that is an analogy used in the Chinese restaurant process. For example, in Fig. 5, we need to assign 8 to any of the topics C1, C2, C3. There is a 3/8 probability that 8 will land in topic C1 topic, 4/8 probability that 8 will land in C2 topic, there is 1/3 probability that 8 will land in topic C3. HDP coherently discovered the topics, like bigger the cluster is the more likely it for the word to join that cluster of topics. It is a good way to choose a fixed set of topics for the LDA model (Wang, Paisley & Blei, 2011). While implementing HDP on our datasets, we test our third approach as well which was manually give the topic number and check the classifier results, to ensure consistency we built around 15 LDA models with different parameters, and compared with HDP results, and choose the best ones, that has a high influence on classifiers results and that gives best classifier results. In the end, the best LDA models were with lemmatized texts (with nouns, adjectives, and verbs), with 100 iterations, and with 10, 15, and 20 topics.

Statistical Validity test

In order to have statistical validity of our model and compare it with a baseline to observe any significant difference in performance, we ran a 5x2cv paired t-test on our dataset. Although there are many statistical tests, we applied this because it is a paired test, and in machine learning this means that the test data for the baseline and the trained model are the same, in our context, it is the same; we used the same Amazon and social media datasets for the baseline and proposed model. As its name implies this test typically split the dataset into two parts (training and testing) and repeat the splitting(50% training and 50% testing) five times, in each iteration (Dietterich, 1998). In each of the five iterations, we fit A and B to the training split and evaluate their performance (pAandpB) on the test split. After this, it again rotates the test and train sets and computes performance again, which results in 2 performance difference measures: (2)${\displaystyle p^1=p_A^1-p_A^1}$ (3)${\displaystyle p^2=p_A^2-p_A^2}$ Then it estimates the estimate mean and variance of differences through following equations;

mean is: (4)${\displaystyle \overline{p}=\frac{p^1+p^2}{2}}$ and variance is: (5)${\displaystyle s^2={\left(p^1-{\overline{p}}^2\right)}^2+{\left(p^2-{\overline{p}}^2\right)}^2}$

The formula of computing t-test statistics for this test is as follows: (6)${\displaystyle t=\frac{p{1}^1}{\sqrt{1∕5\sum _{i=1}^5{S}_i^2}}}$

where p1¹ is p1 from very first iteration. The t statistics assuming that it approximately follows as t distribution with 5 degrees of freedom, and our hypotheses statements and threshold values are;

H0 = Both the classifiers have same performance on this dataset.

H1 = Both classifiers does not have same performance on this dataset.

Our threshold significance level α=0.05 for rejecting the null hypothesis that both classifiers have same performance on this dataset. Under the null hypotheses, t-statistics value approximately follows a t-distribution with 5 degrees of freedom, so its value should remain in a given confidence interval which is 2.571 for 5% threshold, and it indicates that both classifiers have equal performance, if t-statistics value greater than this value, we can reject the null hypotheses. You can implement the 5 by 2 fold cv paired t-test from scratch, but there is a package called MLxtend that implements this test and gives you t-values and p-values of two models (http://rasbt.github.io/mlxtend/user_guide/evaluate/paired_ttest_5x2cv/), in its parameters, we just gave the models names and scoring mode was mean accuracy.

Result and analysis of Amazon dataset

To examine our proposed models based on evaluation measures with different parameters settings, we examined the F1, precision, and recall scores of classifiers.

We randomly divided the data into 5-fold CV, we ran experiments by feeding different LDA topic distributions into the classifiers. The results are in the Figs. 6, 7 and 8, on Y-axis the LDA models with different parameters, and on X-axis it shows the classification results with each classification algorithms. The best performing algorithms (using bigrams and 20 topics) based on the precision scores are MaxEnt, SGD, and SVM. As seen in Fig. 6, these two algorithms (with bigrams features and 20 topic distributions) are slightly better than the other algorithm. In Fig. 7, a noise factor can be seen as MaxEnt and MaxEnt SGD underperformed in terms of recall scores. Then we can see, the model with bigrams and 20 topics achieved the highest F1, precision, and recall score of 91% with support vector machine classifier, while when we apply the trigrams into the LDA model, the MaxEnt classifier algorithm achieved the best result. It implies that the Amazon review dataset has large texts and when you lower the topics then it works well with trigrams and when you increase the topics it works well with bigrams.

Comparison with baseline approaches Amazon dataset

Below in the Table 3 we provided the result without the LDA model, we compare the result with baseline approaches by classifying the data without leveraging the topic distributions. We have implemented the most commonly used TFIDF feature vectors with different classifiers on our Amazon review dataset as a baseline. When we apply the classifiers with TFIDF feature vector representations then F1 scores decrease about 9% and 17% with support vector machine and Multinomial Naive Bayes classifiers respectively as compared to T2F with support vector machine. This means that topic distributions give better results because it semantically capture the words within the documents and their distributions, so that classification performance would be increased, and our proposed framework able to achieve the higher classification results than the baseline approaches. While TFIDF has been popular in its regard, there remains a void where understanding the context of the word was concerned, this is where word embedding techniques such as doc2vec can be utilized, Le & Mikolov (2014). Therefore, we implemented doc2vec with logistic regression classifier as one of our baseline approaches to analyze if it increases performance, the f1 score reaches 86% as compared to the TFIDF approach with SVM classifier which was 82%, but still lower than the F1 score of 91% which we achieved by applying LDA topic distributions as feature vectors.

To examine the classification accuracy and compared it with baseline approaches, we performed 5-fold CV in which we reserved 1/4 observations as the validation set and 4/5 as training observations the advantage of leveraging 5 fold CV is it uses every sample of the dataset in training and testing in iterations. We ran 5-fold CV experiments on the baseline approaches to measure the classification accuracy and also on the proposed model. The detailed accuracy is also shown in Table 3. The comparison of accuracy shown in Fig. 9, starting from fold 1 our approach performs less than the baseline, but after fold 1 it performs better on each fold than the applied baseline and overall average classification accuracy is also higher, the last two columns show the average classification accuracy which improved from 79.3% to 81%, i.e., classification error reduces from 20.7% to 19%. This means that within the dataset with a certain degree of words shared among the documents our framework is capable to reduce the classification error and increase the classification mean accuracy.

To compare the proposed model with the applied baselines approaches and check which approach has more statistical significance on the same Amazon dataset, we ran the 5 X 2 CV paired test on models, and compare the applied baselines with the proposed MaxEnt sgd model, we compared MaxEnt sgd because, if we see the Table 3, the mean accuracy of MaxEnt sgd is higher than other proposed models. We computed the 5X2 CV paired t-test’s t-value and p-value of models, then compare it in the following table. We computed every fold(2 folds) of each iteration(5 iterations) and listed the mean results in the table. You can see in the Table 4, proposed MaxEnt sgd comparison with every baseline model has p-value less than the threshold value α = 0.05, and also t-statistics value is greater than the threshold value, thus we can reject the null hypotheses and accept that two models have significantly different performance, and T2F with MaxEnt sgd with better mean accuracy has performed significantly better than the applied baselines.

Result and analysis of Social media disaster dataset

As in the Amazon dataset, we ran the same LDA models on the social media datasets, and investigate the F1, precision, and recall scores, social media data is more difficult to classify, as it has more slang words, therefore we can see in Fig. 10, the false positive and false negative (recall) is higher than the precision with support vector machine algorithm, in Fig. 11 in precision scores we can investigate rather than with the 20 topics it relatively gives better result with 15 topics and 10 topics with MaxEnt and MaxEnt sgd classifiers.

With both bigrams and trigrams setting and 10 and 15 topics, it performs best with the MaxEnt sgd classifier giving up to 78% F1 score as shown in Fig. 12, it may be because MaxEnt sgd classifier can well handle the noisy short and sparse type of data (Go, Bhayani & Huang, 2009) thus having a higher coverage, and social media is the same kind of noisy unstructured text data. Also in a two-class scenario, it works well because of the binary nature of the target class and we have target class is binary in the social media dataset. Interestingly support vector machines outperformed others algorithms in terms of recall that implies that support vector machines can also somehow, if not at all, handle the noisy data of social media. The social media dataset was short and noisy such as it contains slang, etc. Therefore, it can be seen that the highest F1 score with any parameter setting reached up to 78% as compared to 91% with the Amazon dataset. But that is also satisfactory in the context of the social media data with unsupervised topic modeling. As compared to the Amazon dataset’s large texts, the social media dataset gives more satisfaction with fewer topics, this is because of the length of tweets, which implies that the LDA topic model with less topic setting gives more good results than the LDA topic model with more topics.

Comparison with baseline approaches on social media dataset

In comparison with the baseline approaches that we implemented with different types of word embedding techniques such as TFIDF and doc2vec when applying on social media dataset, it reaches up to 75% high in terms of F1 score with doc2vec embeddings on logistic regression classifier, but still less than the overall highest 78% F1 score with topic distributions as feature vectors, which shows how topic distributions accurately capture the contextual meaning and classify he data accurately. However, an interesting aspect is to analyze that F1, precision, and recall score increases while implementing doc2vec embeddings which indicates among the baseline approaches doc2vec performs best.

As we had run experiments on Amazon datasets same we run 5 fold CV on social media dataset to determine the classification significance by comparing the mean accuracy, although the classification means accuracy drops as compared to when applying on Amazon dataset, still it gives 73% mean accuracy with proposed T2F approach on MaxEnt sgd classifier, when it compared to baseline approaches it falls to 69% with doc2vec feature on MaxEnt classifier, which is highest among the baseline approaches only, but still lower than the proposed T2F approach. The mean accuracy of each fold comparison of highest baseline and highest proposed given in Fig. 13, it starts from fold 1 to fold 5 then, in the end, last two bars showing the mean accuracy which depicts how the classifiers feed with the topic distribution features classified the data significantly better than the baseline approaches and also even better than the mostly used NLP deep learning baseline approach doc2vec.

In order to compare the proposed model with the applied baselines approaches and check which approach has more statistical significance on the same social media dataset, we ran the 5X2 CV paired test on models, and compared the applied baselines with the proposed MaxEnt sgd model; we also compared MaxEnt sgd because also on the social media dataset (see Table 5), the mean accuracy of MaxEnt sgd is higher among the proposed models. We computed the 5X2 cv paired t-test’s t-value and p-value of models, then compare it in the following table. We computed every fold (two folds) of each iteration (five iterations) and listed the mean results in the table. As shown in Table 6, the proposed MaxEnt sgd comparison with every baseline model has p-value less than the threshold value which is α = 0.05, and also t-statistics value is greater than the threshold value, thus we can reject the null hypotheses and accept that two models have significantly different performance, and proposed MaxEnt sgd(T2F) with better mean accuracy has performed significantly better than the applied baselines.

McNemar’s statistical test on unseen data

We did a hypotheses test by applying McNemar’s test to check whether these classifiers are statistically significant on unseen data. In machine learning McNemar’s test can be used to compare the performance accuracy of two models (McNemar, 1947; http://rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar/). McNemar’s test operates on contingency table values that showed in Table 7.

A = SVM classifier

B = MaxEnt sgd classifier

The McNemar’s test is computed as follows: (7)${\displaystyle \frac{{\left(|n01-n10|-1\right)}^2}{n01+n10}}$ n00 = no of samples correctly classified by both A and B

n01 = no of samples correctly classified by A but not by B

n10 = no of samples correctly classified by B but not by A

n11 = no of samples not correctly classified by either A or B

The first step in the statistical test to state the Null hypotheses statement. Our statement is;

H0 = cannot reject the null hypotheses indicating both classifiers have the same performance on the dataset if the calculated p-value greater than the threshold p-value.

H1 = can reject the null hypotheses indicating both classifiers have different performance on dataset if calculated p-value less than threshold p-value.

We ran chi-squared McNemar’s with threshold p-value of 0.05. As shown in Fig. 15 the SVM and MaxEnt with sgd have higher average accuracy than the MaxEnt, so we ran McNemar’s test on these two classifiers and the computed p-value was 0.005667, it indicates that these two classifiers are different, and SVM performs significantly better than the MaxEnt, and the final result is statistically significant.