Feature extraction from customer reviews using enhanced rules

Tools and resources

The experiments conducted in this study were developed and executed using Python, which involved the usage of common Python libraries such as NLTK, NumPy, panda, and CSV. Additionally, Jupiter Notebook was utilized as a developer tool on Windows 10 Pro, whereas model and data libraries from the Flair (https://github.com/flairNLP/flair) framework (Akbik et al., 2019) were employed for POS tagging. The experiment codes (https://github.com/rajeswary3/feature-extraction) (https://doi.org/10.5281/zenodo.8365767) developed for this study are publicly available for reusability purposes.

Dataset

Electronic product review datasets were used in this experimental setup, consisting of five products: two digital cameras, a cell phone, an mp3 player, and a DVD player. These customer review datasets (https://www.cs.uic.edu/ liub/FBS/sentiment-analysis.html#datasets) were developed by Hu & Liu (2004) and are publicly available. Further descriptive details of these datasets are provided in Table 2. Mainly, these datasets were chosen as they fit the research scope for the following reasons:

• They are open datasets.

• They were extracted from review domains and are not based on microblogs or Twitter.

• They have been used in many recent related studies (refer to Table 1).

• Opinionated product features are more tangible in nature, suiting explicit extraction.

In order to comprehend the actual performance of the proposed approach, it is important to compare results from studies that use the same product review datasets and feature extraction technique. The main reason for limiting the research scope as such is to prove the performance is being fairly compared in the same controlled environment that executes on the same product domain datasets with same size and structure.

The review documents are in a structured format, with features, polarity scores, and types (implicit/explicit) for each review sentence labelled. The annotation for each sentence starts either with [t] to mark the review title or labeled features followed by “##” for review sentences. The features were labeled with positive or negative scores within the range of +3 for the strongest opinion and −3 for the weakest opinion. Implicit features, where the features were not present in the review sentences, were tagged [u] and [p]. Additional tags that may be useful for future studies, such as [s] for author suggestions and [cs] for product comparisons, were also part of the review documents. As this study focuses on explicit feature extraction, only relevant review sentences were extracted through preprocessing. Therefore, of the 3,945 review sentences examined, 1,726 reviews with labeled features were selected and processed.

A combination of existing rules and new rules was tested with an opinion lexicon and two different POS taggers. The scope of this research is to identify explicit features in opinionated content; thus, it is compulsory for a feature to be accompanied by an opinion, irrespective of the polarization of the opinion words. For results evaluation, both labeled and predicted features were used.

Preprocessing stage

Preprocessing constitutes the first stage of this study’s proposed feature extraction methodology. As shown in Fig. 1, the preprocessing stage begins with the review documents being processed to extract only relevant review sentences that contain explicit features. For the preprocessing activity, only review contents were used, hence review titles and implicit features related sentences were removed as the scope of the research is to identify explicit features only. The objective of this omission is to improve the performance of explicit feature extraction.

Line [1] is tagged as title [t] of the review. Lines [2] and[3] are the contents of the review. However, in Line [2] , there are no features annotated by the author. Line [3] has “battery” as the feature. Hence Line [1] and Line [2] are removed during preprocessing.

Review sentences composed by customers in plain language typically reflect their preferences, opinions, prior expectations, and emotions regarding the purchase and usage of a product or service. These sentences may also contain noise, spelling errors, grammar mistakes, abbreviations, unwanted expressions, or extraneous information (Haddi, Liu & Shi, 2013). Thus, text cleaning, as part of preprocessing, is essential in ensuring a high-quality processing. Text cleaning involves removing spaces, special characters, and stop words, as well as stemming and lemmatization.

Accordingly, the review sentences underwent a cleaning process aimed at rectifying spelling errors. Correcting misspelled words is crucial, as inaccuracies in spelling can result in a loss of intended meaning and yield unreliable outcomes. Reference lists (http://introcs.cs.princeton.edu/java/44st/misspellings.txt) from a prior study (Krishnakumari & Sivasankar, 2018) were used as samples of misspelt words for correction. Another step in text cleaning is to remove stop words and special words. Stop words were removed based on the NLTK stop word (https://www.nltk.org/nltk_data/) library for the English language. Punctuations and special characters, such as “@” or “&”, were also eliminated as part of the cleaning exercise. Following this, lemmatization was performed on words in the sentences to identify their root terms.

As the final step in preprocessing, each review sentence in the document was tagged with POS labels. Specifically, singular and plural nouns were identified as NN/NNS, adjectives as JJ, various types of adverbs as AVB/RB/RBR/RBS, different types of verbs as VB/VBZ/VBD, prepositions as IN, determiners as DT, opinion words as O, and features as A. Various POS libraries are used in feature extraction-related studies to determine the impact of tagging on the accuracy of opinionated feature extraction. The NLTK (https://www.nltk.org/) POS Tagger and Stanford (https://nlp.stanford.edu/software/tagger.html POS Tagger are popular POS tagger libraries, while the Flair (https://github.com/flairNLP) framework is considered a simple-to-use tagger. This study initially used both the NLTK and Flair tagger libraries in Python. Based on the results discussed in ‘Results and Discussion’, the Flair framework was selected for the final analysis.

Feature extraction stage

This section provides a comprehensive discussion of the processes and pattern rules utilized in this study. The second stage of processing involved feature extraction using a combination of rules employed in prior studies and newly devised rules, aimed at enhancing opinionated feature extraction results.

One of the initial opinion lexicons was introduced by Hu & Liu (2004). It has evolved and been continually updated since 2004, now comprising approximately 6,800 positive and negative words. To complement this lexicon, this study also utilized another lexicon authored by Almatarneh & Gamallo (2018). This supplementary lexicon (https://github.com/almatarneh/LEXICONS) contains extreme opinions that represent the most favorable or unfavorable assessments. Consequently, extreme words not found in Hu & Liu’s (2004) lexicon were adopted from the lexicon by Almatarneh & Gamallo (2018). These lexicons serve as valuable resources for validating the extracted opinion words.

Selected existing rules from prior studies have been identified and presented in Table 3. To achieve this study’s objective of improving feature extraction efficiency, additional rules were formulated to extract more precise features. Previous studies have predominantly focused on noun-based words as features, despite evidence that certain features are non-noun words, such as verbs. This means features can indeed be extracted from non-noun words, and opinions need not exclusively consist of adjectives. As such, new rules were designed in this study to address the issue of some features being overlooked due to not falling within the common noun category. While nouns are commonly associated with features and adjectives with opinions, the new rule set acknowledges the presence of outliers. Table 4 provides details on the new rules that were formulated based on an analysis of sample datasets and observation techniques, while Algorithm 1 outlines the steps used for extracting features. Further experiments were conducted on robust datasets to validate the newly developed rules. All the codes developed for the experiment are publicly available on the authors’ GitHub (https://github.com/rajeswary3/feature-extraction) (https://doi.org/10.5281/zenodo.8365767).

Evaluation stage

In this study, the evaluation criteria were set in reference to Liu et al. (2016), whereby precision, recall, and F-measure were calculated using TP (true positive), FP (false positive), and FN (false negative) values. Precision denotes the percentage of correctly identified features out of the total identified features, whereas recall identifies the percentage of identified features out of the total labelled features. The analysis of extraction is based on Table 5.

Precision, recall, and F-measure calculations (Liu et al., 2016) were formulated as Eqs. (1), (2) and (3) below: (1)${\displaystyle \text{Precision},\mathrm{p}=\frac{\text{Extracted features}\cap \text{Labelled features}}{\text{Extracted features}}}$ (2)${\displaystyle \text{Recall},\mathrm{r}=\frac{\text{Extracted features}\cap \text{Labelled features}}{\text{Labelled features}}}$ (3)${\displaystyle \mathrm{F}-\text{Measure}=\frac{2\mathrm{pr}}{\mathrm{p}+\mathrm{r}}.}$

The experiments were conducted on five different datasets, and performance was individually assessed before calculating the average results for the overall product datasets. Dataset details are provided in Table 2.

POS tagging framework selection

To determine the most suitable POS tagging framework for this proposed method, experiments were conducted on each of the individual products to evaluate the performance of two different tagging approaches: NLTK tagging and Flair tagging. The graphs below illustrate the results of precision, recall, and F-measure, demonstrating that the Flair tagging framework outperformed NLTK tagging. The Flair framework is a modern open-source NLP library built on PyTorch. It consistently exhibited superior performance to the NLTK library when applied to various product review documents. Therefore, subsequent feature extraction experiments utilizing rules were executed using the Flair framework.

Feature extraction

Table 6 presents the overall performance of the proposed feature extraction method. Feature extraction experiments involved two sets of rules. The first set of rules consisted of existing rules used in prior studies, as presented in Table 3. The second set of rules combined the existing rule sets (Table 3) with a new set of rules (Table 4) formulated in this study. Table 7 showcases the results obtained using both sets of rules. It is evident that average precision improved by 6%, average recall by 6%, and the average F-measure value by 5% with the inclusion of the new rules introduced in this study. These new rules led to the identification of more features, directly enhancing the precision and recall of the study. The superior performance results in terms of precision and recall are attributed to the high data quality, which is a result of extensive data cleansing during the preprocessing stage.

Further comparisons were made with previous studies that used rules for feature extraction on the same datasets as this study. Performance comparisons were made based on precision, recall, and F1-measure, as presented in Table 8. Based on the results, it can be observed that this study has set a new standard for pattern-based studies in terms of the number of rules and performance achieved. This study notably achieved the highest precision and F-measure for explicit feature extraction among all past studies.

Table 9 provides a detailed comparison of the performance for each dataset used in this study against the study that attained the highest performance in the past. While Tubishat, Idris & Abushariah (2021) achieved the highest values since publication in 2021, the subsequent study by Tran, Duangsuwan & Wettayaprasit (2021) reported lower results. An important aspect to highlight here is that Tubishat, Idris & Abushariah (2021) employed additional tasks and resources, such as the product manual and optimization techniques, to enhance feature extraction performance. Although they mentioned conducting pruning based on the product manual, there is limited clarity on how the product manual or user guide was used in the research to detect product features. Detecting feature frequency with a certain threshold (e.g., a threshold of two for single-word features) within the product manual may appear ambiguous when considering the entire guide is to be interpreted and processed word by word. Attempts to contact the authors for clarification were unsuccessful.

Further to that, despite the successful implementation of optimization techniques in Tubishat, Idris & Abushariah’s (2021) research, this study did not consider optimization techniques for several reasons. According to Osaba et al. (2021), optimization needs to be evaluated based on various criteria, including processing time, memory requirements, and the time needed to obtain results, particularly in an environment with significant computational demands. Tubishat, Idris & Abushariah (2021) aimed to optimize and select the best subset of rules out of 126 rules defined using a training dataset. As a result of their LSA algorithm, a set of 57 rules was frequently selected as the best. Hence, their decision to utilize an optimization algorithm was justified. In contrast, in this study, only a set of 41 rules (both existing and new rules) was used. The decision not to employ optimization in this study thus saved memory and processing time.

Lastly, when Tubishat, Idris & Abushariah (2021) applied rules without optimization techniques, they achieved a precision of 0.75 and an F-measure of 0.84. Comparatively, the experiment results from this study demonstrated higher precision than all baseline studies, indicating that the study successfully achieved its research objective. It is also established that features for extraction are not necessarily limited to nouns, and opinions are not solely adjectives. They can encompass other linguistic categories as long as they reflect a specific property or characteristic of a product or service.

An extensive review was made of all the recent studies involving pattern-based feature extraction. This is to identify the gaps that could be addressed by the proposed approach. The review of all the recent related studies (related to rule-based extraction) is crucial to ensure there is no repetitive effort in the same field/technique. These findings have provided a valid problem statement that is very much focused on rule-based feature extraction. There are a few reasons why no comparison is made to other types of alternative techniques.

1. Firstly, the scope of research was focused on rule-based techniques to ensure that the approach can provide a solution in improving feature extraction performance by addressing a problem statement within the field/technique. Hence all the results evaluations were made against research that also used rule-based techniques.

2. The research objective is to improve the feature extraction performance, where undoubtedly the improvisation is more justifiable when comparing results and performance produced by different studies within the same field. This study does not mean to identify which technique yields better performance.

3. Alternate techniques, such as deep learning or improved optimization techniques may require larger data points and training datasets, increased setup cost and time, powerful hardware setup and complex algorithms. Comparing extraction performance produced from such a complex and significant computational setup is not fairly justified. Results produced by such deep learning methods are probably much better upon execution, however, the investment in setting up a proper environment for execution is comparatively huge. As a reference to Tubishat, Idris & Abushariah (2021), who have used optimization on top of rule-based mining techniques have better results upon optimization. However, this resulted in more memory and time consumption.

4. In addition to (c), evaluating the performance produced with different techniques, which were executed under different experimental setups and parameters, may not be a fair comparison.

According to the Kano model, businesses must revolutionize their strategy in identifying customer needs, determining essential product features to meet those needs, and rectifying product deficiencies. This transformation necessitates a shift in how customer feedback is collected and analyzed, moving away from the traditional approach of waiting for customer complaints or using formal feedback channels. Rather, customer reviews serve as a crucial marketing tool, attracting more customers and aiding newcomers in making informed decisions about product performance. An effective evolutionary approach is to leverage computational assistance in mining customer reviews. The challenge, however, lies in identifying relevant product features within these reviews. This problem has prompted numerous researchers to seek ways to detect as many product features as possible in review documents. To this end, this study’s proposed approach represents a successful attempt to identify features that attract customers but have been undiscovered by existing rules.

Many customer review management tools, such as Trustpilot or feefo.com, are available in the current market, assisting both businesses and customers. Similar tools can be developed to assist customers using the enhanced rules proposed in this study. Identifying accurately reviewed features will further enhance search engine optimization (SEO) credibility and, ultimately, foster customer trust in making informed decisions. Simultaneously, regression analysis can be conducted on the extracted features, particularly in terms of the correlation and dependence among them, to provide more impactful insights to customers. For instance, one can explore the relationship between the price and size of a hotel room, offering customers a better understanding of whether a higher price is justified when the room size is larger. Notably, the proposed approach in this study ensures a higher number of features are extracted, which is crucial to comprehensively assess the dependency and relationships between features.

Nevertheless, the challenge in implementing this proposed solution lies in the thorough preprocessing it requires. Review data cannot be used directly after extraction from review sites; instead, preprocessing activities, including noise removal and spelling error correction, are essential to present a clean review document for further processing. Additionally, as demonstrated in Figs. 2 to 4, the choice of different POS tagging frameworks may impact extraction performance. Hence, for the real-world implementation of this proposed approach, using the Flair framework for POS tagging is highly recommended.