Towards an automated classification phase in the software maintenance process using decision tree

Sorting maintenance requests

Machine learning techniques have been used to sort and categorize MRs. Phetrungnapha & Senivongse (2019) developed an approach to categorize user reviews into feature requests or bug reports using several machine learning techniques, including DT, linear SVC, KNN, NB, logistic regression, and ensemble methods. Ciurumelea, Panichella & Gall (2018) introduced the AUREA tool to help developers analyze user feedback and plan maintenance activities. Ekanata & Budi (2018) used machine learning to categorize user reviews automatically, finding logistic regression to be the most effective. Levin & Yehudai (2017) classified commitments in maintenance activities using J48, GBM, and RF, achieving an accuracy of 76%.

Otoom, Al-jdaeh Hammad & Hammad (2019) built a classifier to distinguish software bug reports into corrective or perfective reports using keyword frequency and classification algorithms, achieving an average accuracy of 93.1% with SVM. Pandey et al. (2017) analyzed bug reports using various algorithms, including RF and SVM, finding high performance with RFs and SVMs. Ahmed, Bawany & Shamsi (2021) introduced the CaPBug framework, using NLP and machine learning to classify and prioritize error reports, achieving class prediction accuracy of 88.78% with RF.

Ranking maintenance requests

Researchers have also addressed ranking MRs using machine learning techniques. Srewuttanapitikul & Muengchaisri (2016) proposed prioritizing software flaws based on severity, priority, and user reports. Guzman, Ibrahim & Glinz (2017) surveyed researchers and practitioners to rank and prioritize tweets for software development. Ekanayake (2021) proposed using the RAKE algorithm for keyword extraction and NB, DT, and logistic regression for prioritizing error reports. Alenezi & Banitaan (2013) predicted bug priority using DTs, NB, and RF, concluding that DTs and RFs outperformed NB.

Researchers have explored various machine learning techniques for ranking maintenance requests based on severity and priority. Srewuttanapitikul & Muengchaisri (2016) suggested using a natural language processing approach combined with an analytic hierarchy process to prioritize MRs by severity, priority, and the number of users reporting the same defect. Guzman, Ibrahim & Glinz (2017) conducted a survey of 84 software engineering practitioners to rank tweets for software development, emphasizing the importance of audience-based requirements engineering. Ahmed, Bawany & Shamsi (2021) introduced the CaPBug framework, which uses natural language processing and machine learning to classify and prioritize error reports into six categories and five priority levels. The CaPBug framework achieved class prediction accuracy of 88.78% with the RF classifier and priority prediction accuracy of 90.43%.

Ekanayake (2021) proposed a model using the RAKE algorithm to extract keywords from error reports, converting them into attributes for prioritizing MRs with NB, DT, and logistic regression. The model achieved logistic regression accuracy of 0.86, with DT and NB accuracies of 0.81 and 0.79, respectively. Alenezi & Banitaan (2013) suggested using machine learning algorithms to predict bug priority, concluding that DTs and RFs outperformed NB. Tian et al. (2015) proposed a multi-factor analysis approach for predicting bug report priority, achieving a relative improvement of 209% in the average F-measure. Umer, Liu & Sultan (2018) introduced an emotion-based approach for predicting bug report priority using natural language processing to identify emotional words and a supervised machine learning classifier. Their approach outperformed the latest technologies, improving the F-measure by more than 6%. Ramay et al. (2019) proposed a deep neural network-based approach for predicting bug report severity, which outperformed existing methods and improved the F-measure by 7.90%.

Accepting/rejecting maintenance requests

Various machine learning techniques have been used to predict the acceptance or rejection of MRs. Nizamani et al. (2018) proposed a NB polynomial approach for this task, achieving an accuracy of 89.25%. Umer, Liu & Sultan (2019) used sentiment analysis to predict approval, achieving 77.90% accuracy. Cheng et al. (2021) used deep learning for approval prediction, achieving 90.56% accuracy. Nyamawe et al. (2020) recommended refactorings based on feature requests, achieving 83.19% accuracy. Nizamani et al. (2018) proposed a NB polynomial approach for predicting the acceptance or rejection of improvement requests, using data from Bugzilla. Their method achieved an accuracy of 89.25%. Umer, Liu & Sultan (2019) developed a sentiment-based approach to predict the approval of enhancement reports, achieving 77.90% accuracy and a significant improvement in the F-measure to 74.53%.

Cheng et al. (2021) proposed a deep learning approach to predict the approval of enhancement reports, achieving 90.56% accuracy, 80.10% recall, and 85.01% F-measure. Nafees & Rehman (2021) used SVM for predicting the acceptance or rejection of improvement reports, comparing it with logistic regression and multinomial NB. Their study found that SVM outperformed other algorithms with high accuracy. Nyamawe et al. (2020) suggested a machine learning approach for predicting software refactorings based on feature requests, achieving 83.19% accuracy. Arshad et al. (2021) proposed a deep learning technique for predicting the resolution of enhancement reports, using Word2Vec and a deep learning classifier to learn semantic and grammatical relations between words. Their approach enhanced performance and demonstrated effective prediction accuracy.

Collecting data

To collect data, we contacted several researchers and developers, seeking a dataset consisting of MR texts and their sorting, ranking, and acceptance/rejection classes. Unfortunately, we did not find a suitable dataset. Therefore, we turned to online data collection sites, such as Mendeley Data, which provide datasets used in scientific research. We used two datasets: the Commit dataset from Levin & Yehudai (2017) and the Pan dataset from Al-Hawari, Najadat & Shatnawi (2021). The Commit dataset classifies MRs as perfective, adaptive, and corrective, while the Pan dataset categorizes user reviews into problem discovery, feature request, information seeking, and information giving. We reclassified the Pan dataset to match the Commit dataset’s categories. The final dataset consisted of 1,656 MRs, as shown in Fig. 3. The demography of MRs is shown in Table 1. We split the dataset into 80% for training and 20% for testing, following common recommendations (Joseph, 2022; Rácz, Bajusz & Héberger, 2021).

Processing data

Before applying DTs, we processed the natural language to convert unstructured data into structured data. We removed links, user/application information, numbers, stop words, and punctuation. We then applied lemmatization to retrieve words in their normalized form, as shown in Fig. 4 (Razno, 2019; Korenius et al., 2004). The Natural Language Tool Kit (NLTK) library was used for lemmatization (Bird, 2006).

Extracting features

We used the term frequency-inverse document frequency (TF-IDF) technique for text feature extraction, which measures document relationships (Zhang, Zhou & Yao, 2020; Qaiser & Ali, 2018). The TF-IDF vectorizer library was used to extract features from the processed dataset.

Decision tree

We developed the DT using Python, which is suitable for machine learning and AI. We used the Scikit-learn library for supervised learning algorithms and statistical analysis (Pedregosa et al., 2011). The dataset was divided into 80% for training and 20% for testing. The DT was trained to sort, rank, and accept/reject MRs. We used the fit() method to train the model, taking the result of feature extraction (TF-IDF) and the desired column in the trained dataset. The workflow of the implemented DT is shown in the Fig. 5. The DT classified MRs into three classes: perfective, corrective, and adaptive. The model was trained for ranking MRs based on severity and priority and for accepting/rejecting MRs.

Result of sorting MRs

When we applied the DT to sort MRs, as shown in Fig. 6, the accuracy was 71.08%, precision was 71.1%, and recall was 71.1%. The confusion matrix of sorting MRs is represented in Fig. 7. By using actual and predicted values, we found that 42 MRs were correctly classified as adaptive, 136 as corrective, and 58 as perfective. The adaptive class represented 20.5% of the actual and 19.9% of the predicted classes, the perfective class represented 28.6% of the actual and 30.1% of the predicted classes, and the corrective class represented 50.9% of the actual and 50% of the predicted classes, as shown in Fig. 8.

Result of ranking MRs

When we applied the DT to rank MRs, as shown in Fig. 9, the accuracy was 50%, precision was 50%, and recall was 50%. The confusion matrix of ranking MRs is shown in Fig. 10. Using actual and predicted values, we obtained 35 MRs correctly classified as class 1, 11 as class 2, 29 as class 3, and 91 as class 4. Class 1 represented 22.6% of actual and 20.5% of predicted classes, class 2 represented 13.9% of actual and 10.8% of predicted classes, class 3 represented 23.8% of actual and 19.9% of predicted classes, and class 4 represented 39.8% of actual and 48.8% of predicted classes, as shown in Fig. 11.

Result of accepting/rejecting MRs

When we applied the DT to accept or reject MRs, as shown in Fig. 12, the accuracy was 64.15%, precision was 64.2%, and recall was 64.2%. The confusion matrix of accepting or rejecting MRs is shown in Fig. 13. By using actual and predicted values, we obtained 149 accepted MRs and 64 rejected MRs. The accepted MRs represented 66.6% of the actual and 59% of the predicted classes, while the rejected MRs represented 33.4% of the actual and 41% of the predicted classes, as shown in Fig. 14.