Enhancing software defect prediction: a framework with improved feature selection and ensemble machine learning

Motivation of the study

Software quality assurance (SQA) aims to ensure software quality throughout its development life cycle. A critical SQA task is the early identification of defective modules, as addressing defects in later stages is resource-intensive (Iqbal et al., 2019b). Effective defect prediction relies on optimal feature selection from historical software data. Previous studies have demonstrated that feature selection techniques can enhance classifier accuracy (Balogun et al., 2021; Singh & Haider, 2022). However, classification techniques alone have not consistently delivered exceptional results. By integrating diverse classification techniques, Ensemble learning crucially enhances predictive accuracy and robustness (Jacob et al., 2021; Ali et al., 2020; Alazba & Aljamaan, 2022). Recent research (Iqbal & Aftab, 2020) in software defect prediction using machine learning classifiers; has revealed varying accuracy rates—ranging from 79.59% for CM1 to 74.8% for PC4—indicating suboptimal outcomes in some instances (62.78% for JM1, 62.16% for MC2, 77.33% for MW1, 89.65% for PC1, and 75.94% for PC3). Therefore, there is a demand for a framework that combines feature selection via genetic algorithms with heterogeneous classification methods like RF, SVM, and NB, along with a voting ensemble approach, to achieve higher accuracy.

Organization of the study

This study has been organized as follows: ‘Literature Review’ provides an overview of existing research in the field through the literature review. ‘Materials and Methods’ outlines the proposed framework, with detailed descriptions of its stages. ‘Results and Discussion’ encompasses an extensive analysis and discussion of the results obtained by applying the proposed framework. ‘Threats to Validity’ discusses the potential validity concerns associated with the research. Finally, ‘Conclusion’ concludes the study by concisely summarizing the findings and directions for future research.

Feature selection

Regarding the prediction of software defects, a hybrid framework using various classification algorithms was developed by implementing twelve NASA datasets (Iqbal et al., 2019b). The experiment had two individual approaches; the first was with feature selection, and the other was without feature selection. Each approach implemented two ensemble strategies, i.e., bagging and boosting, taking RF as the base classifier, improving the framework’s accuracy. Similarly, a researcher in Balogun et al. (2019a) explored filter feature ranking along with fourteen filter subset selection (FSS) techniques with five NASA datasets and best first search (BFS) as an FS technique. They concluded that the FS strategy generates results with better prediction accuracy and that the feature filter ranking methods were more stable for prediction purposes. Another researcher worked on FS and studied eight FS techniques using five supervised and five unsupervised learning models to cross-check the variance in performance. Both learning models performed better when implemented with the FS technique. They observed that the results of neural network-based techniques are better for unsupervised learning models, while consistency & and correlation-based FS methods work better for supervised learning models  (Kondo et al., 2019). There are several ML techniques for FS; among them, the Genetic Algorithm (GA) is a robust algorithm used to solve optimization problems (Ayon, 2019). This algorithm is closer to nature, is based on the natural process of evolution, and is efficient in solving computationally expensive problems (Hamdia, Zhuang & Rabczuk, 2021). A recent study reviewed genetic algorithms and their variants, mainly focusing on multimedia and wireless network applications (Katoch, Chauhan & Kumar, 2021). Another FS technique using a Genetic search and encoder–decoder (E–D) model having long short-term memory (LSTM) was implemented to forecast air pollution particulate matter (PM) 2.5 using datasets taken from Hanoi and Taiwan. The E–D model generated results showing improved accuracy (Nguyen et al., 2021).

Classification

Classification, a supervised machine learning technique, is widely applied in various prediction models, including weather forecasting, disease prediction, sentiment analysis, and software defect prediction  (Iqbal et al., 2019b; Luo et al., 2022; Li, Ortegas & White, 2023). In software defect prediction (SDP), a comparative analysis of four classifiers was conducted using NASA’s datasets (Daoud et al., 2022). The fused artificial neural network Bayesian regularization (ANN-BR) classifier, particularly the Bayesian regularization (BR) classifier, demonstrated remarkable performance. Iqbal et al. (2019a) expanded this research by analyzing twelve datasets from NASA, employing multiple classifiers, and comparing results using various performance measures. Across different datasets and algorithms, significant variations in classification performance were observed. For example, the JM1 dataset excelled when paired with the RBF algorithm, while KC3 demonstrated improved results with the MLP algorithm. MC1 achieved remarkable results using KStar, and PC2 also performed well with KStar. In another study, ten classifiers were evaluated for SDP, with the random forest classifier showing enhanced performance on the PROMISE datasets (Cetiner & Sahingoz, 2020). Additionally, a software defect prediction model utilizing LASSO-SVM with a reduced-dimension dataset was proposed, incorporating cross-validation to enhance model performance (Wang et al., 2021). In Iqbal (2019), researchers developed a model using artificial neural networks and conducted an empirical comparison of backpropagation training algorithms, including Liebenberg-Marquardt, Bayesian Regularization, Scaled Conjugate Gradient, and BFGS Quasi-Newton techniques, on CM1, KC3, MW1, PC1, and PC2 datasets from NASA and remarkable accuracies were achieved on all the employed datasets.

Ensemble learning

Ensemble learning (EL) is a technique in ML in which predictions from several weak classifiers are integrated to produce a strong classifier that generates better results than standalone classifiers (Mehta & Patnaik, 2021). EL offers a range of homogeneous classifiers like bagging, boosting, rotational forest, etc., and several heterogeneous classifiers, including voting, stacking, etc. (Wu & Wang, 2023). Voting is a heterogeneous ensemble classifier that combines predictions from diverse base classifiers. A software defect prediction framework was proposed by a researcher in Javed (2021) using a nested EL technique, i.e., voting as the master classifier, and three base ensemble classifiers, i.e., bagging, boosting, and stacking. The accuracy produced by the proposed framework on two different NASA datasets was 83.46% and 79.65%. Moreover, a software defect prediction model was developed by a researcher in Matloob (2020) using multi-layer feed-forward neural networks and stacking as an ensemble technique. Six search methods were implemented for feature selection and multilayer perceptron was used as a subset evaluator. The accuracy achieved on NASA’s datasets using best-first search, greedy stepwise search, and genetic search was 80%, 75%, and 76% respectively.

Limitations of previous research

The existing studies in software defect prediction have made noteworthy contributions; however, several limitations have been identified, particularly low accuracy. Authors in Amin (2019) implemented iterative feature selection using SMOTE and BORUTA approaches with SVM and neural networks as classifiers, achieving a maximum accuracy of 76% on NASA’s datasets. However, the absence of ensemble learning techniques in the methodology is notable, potentially introducing bias and low accuracy, limiting the overall predictive capabilities. Iqbal et al. (2019a) analyzed twelve datasets from NASA, employing various classifiers such as naïve Bayes, multilayer perceptron (MLP), radial basis function (RBF), SVM, K nearest neighbor (KNN), kStar, OneR, PART, decision tree (DT), and random forest (RF). While showcasing diverse classifiers, the study lacked the incorporation of feature selection and ensemble learning methods. This absence may hinder the model’s generalization of datasets, impacting predictive accuracy. Similarly, the authors in Iqbal (2019) developed a model using artificial neural networks, comparing backpropagation algorithms empirically. Despite achieving notable accuracies ranging from 81.70% to 86.85% on different NASA datasets, the study lacked the incorporation of feature selection and ensemble learning methods with a combination of diverse classifiers. This limitation may affect the model’s robustness and generalization to diverse software defect scenarios.

To address these limitations, a framework is proposed for software defect prediction that implements a genetic algorithm to perform feature selection and voting as an ensemble learning technique with a heterogeneous combination of RF, SVM, and NB as base classifiers for enhancing the prediction power of software defect prediction.

The summary of the literature review is presented in Table 1. It outlines the techniques proposed for SDP, the source of datasets used for experimentation, the specific datasets employed, and the performance measures employed to analyze the results.

Dataset selection

This research utilizes seven NASA datasets, namely CM1, JM1, MC2, MW1, PC1, PC3, and PC4 which are publicly accessible. These datasets were collected from NASA’s real software projects and developed in different languages. In Shepperd et al. (2013), researchers analyzed NASA’s defect datasets and produced two individually cleaned versions of the datasets, i.e., DS’ and DS”, the benchmark datasets. The DS dataset contains identical and conflicting values. Meanwhile, D excludes duplicate and inconsistent data. This research uses seven datasets from the DS” version to implement the proposed framework. Table 2 presents the cleaning criteria used for NASA datasets implemented by Shepperd et al. (2013).

Each dataset consists of many independent attributes, i.e., BRANCH_COUNT, CYCLOMATIC_COMPLEXITY, HALSTEAD_EFFORT, LOC_TOTAL, LOC_EXECUTABLE, etc., and one dependent attribute called the target class. The dependent attribute is predicted based on the independent attributes. The target class contains a Boolean value “Y” or “N” where “Y” represents that the particular module is defective and “N” represents that it is non–defective. The dataset was partitioned into training and testing sets following a 70–30 proportion, employing a class-based splitting rule to ensure a representative distribution across classes. The details of the datasets used in this research are given in Table 3.

The target class distribution of the employed datasets is presented in Fig. 2.

Feature selection

The software consists of many features, but only a few features positively impact the performance of classifiers (Kondo et al., 2019; Ali et al., 2023). In this research, a filter-based feature selection technique has been used that evaluates all the attributes available in the underlying dataset and draws only those attributes that are more appropriate based on the target class. The feature selection technique comprised two parts, i.e., (1) the search method and (2) the attribute evaluation method. This research employs a GA search method and CfsSubsetEval as an attribute evaluation method. GA involves stages like population initialization, fitness function calculation, selection of fittest individuals, crossover, mutation, and termination upon reaching a defined threshold. GA was incorporated with the following parameter values: a population size of 20 individuals, a maximum of 20 generations, a crossover probability set to 0.6, and a mutation probability of 0.033. These parameters were precisely chosen through empirical testing and iterative tuning to optimize the effectiveness and efficiency of the feature selection process. GA’s evolutionary approach allows it to efficiently explore diverse feature subsets, aligning well with the feature selection problem. Its capability to handle non-linear relationships within the dataset is particularly advantageous, contributing to identifying informative features while mitigating computational costs (Bindu & Sabu, 2020; Maleki, Zeinali & Niaki, 2021). The adaptability of GA to problem complexity adds a robust dimension to the feature selection process, enhancing the overall predictive power of the framework (Hamdia, Zhuang & Rabczuk, 2021). Hence, the algorithm identifies and returns the best-performing individuals crucial for problem-solving (Peng et al., 2021). This research employs the Correlation-based Feature Selection (CFS) method, a filter technique for evaluating each attribute’s predictive power concerning the output class. Attributes with the highest correlation to the output class are selected while minimizing inter-correlation among dataset attributes  (Zhu et al., 2021). GA-driven feature selection identified an optimal subset of features, ensuring that the most informative attributes were retained through an iterative selection process, crossover, and mutation. This process dynamically explored the solution space to converge upon subsets that maximize the classifier’s predictive accuracy. The graphical representation of selecting the most optimal features using a genetic algorithm is shown in Fig. 3.

The features produced after applying the genetic algorithm are listed in Table 4.

A thorough feature correlation analysis was conducted for subsequent analysis to gain deeper insights into the dataset and understand the relationships among different features. This analysis is crucial in revealing potential dependencies or redundancies among the selected features. The feature selection process, which leveraged a genetic algorithm search method in conjunction with the CfsSubsetEval evaluator, aimed to identify the optimal subset of features for the given task. Before showcasing the results of this feature selection process, examining the degree of correlation between the features themselves is imperative. Correlation diagrams visually represent these relationships and play a pivotal role in the decision-making process when selecting our predictive model’s final set of features. By unveiling feature correlations, this analysis ensures that the selected features are both informative and independent, thus contributing to the overall effectiveness of our software defect prediction framework. The correlation graph for each training dataset is presented in Fig. 4, which uses color intensity to represent the strength and direction of feature correlations. Darker shades indicate stronger correlations, with positive correlations represented by warmer and negative correlations by cooler colors.

Classification

This research implies three classification algorithms of heterogeneous nature, i.e., RF, SVM, and NB, that produced significant results in the development of a software defect prediction framework. The description of each classifier is given as follows:

Random forest (RF): RF utilizes the strength of decision trees, where each tree is trained on a subset of the dataset and makes independent predictions. The final output is determined through a voting mechanism, providing a more robust and accurate prediction  (Soe, Santosa & Hartanto, 2018; Ibrahim, Ghnemat & Hudaib, 2017). This approach enhances predictive accuracy and fosters resilience against overfitting and variability in software defect patterns (Alshammari, 2022; Mafarja et al., 2023). In mathematical terms, we can describe RF as an ensemble of decision trees: (1)${\displaystyle \mathrm{RF}\left(\mathrm{X}\right)=\text{Tree}\text{\_}1\left(\mathrm{X}\right)+\text{Tree}\text{\_}2\left(\mathrm{X}\right)+\cdots +\text{Tree}\text{\_}\mathrm{N}\left(\mathrm{X}\right).}$

where RF(X) represents the Random Forest’s prediction for input data point X. Tree_1(X), Tree_2(X), …, Tree_N(X) represent the predictions of individual decision trees in the forest. Each tree _i (X) is the output of an individual decision tree, which can be a complex, recursive structure involving feature selection, node splitting, and class prediction.

Support vector machine (SVM): SVM operates by identifying a hyperplane that best separates different classes in the feature space, maximizing the margin between them (Wang et al., 2021; Kumar & Singh, 2017). By leveraging the principles of margin maximization, SVM enhances the framework’s ability to distinguish complicated patterns, providing a valuable addition to accurate software defect prediction (Mustaqeem & Saqib, 2021). SVM classifier can be represented in mathematical form as follows:

Given a dataset with feature vectors x_i (i = 1, 2, …, N) and corresponding binary labels y_i (y_i ∈ {-1, 1}), the goal of SVM is to find a decision boundary in the feature space that maximizes the margin between the two classes while minimizing classification errors (Husin, Pribadi & Yohannes, 2022).

The decision boundary can be defined as: (2)${\displaystyle \mathrm{f}\left(\mathrm{x}\right)=\text{sign}\left(\sum \left[\text{alpha}\text{\_}\mathrm{i}.\mathrm{y}\text{\_}\mathrm{i}\left(\mathrm{x}\text{\_}\mathrm{i}.\mathrm{x}\right)+\mathrm{b}\right]\right).}$

where f(x) is the decision boundary function for classifying a new data point x, alpha_i represents the Lagrange multipliers associated with each data point, x_i represents the support vectors (data points that lie on the margin or are misclassified), x⋅x_i represents the dot product between the feature vectors x and x_i, b is a constant bias term.

In the case of a polynomial kernel of degree 2 (quadratic), the kernel function K(x, x_i) is defined as: (3)${\displaystyle \mathrm{k}\left(\mathrm{x},{\mathrm{x}}_{\mathrm{i}}\right)=\left(\mathrm{x}.{\mathrm{x}}_{\mathrm{i}}+1\right)\stackrel{ˆ}{}2.}$

So, the decision boundary for the SVM with a polynomial kernel of degree 2 (C = 2) would involve using this kernel function within the decision boundary equation: (4)${\displaystyle \mathrm{f}\left(\mathrm{x}\right)=\text{sign}\left(\sum \left[\text{alpha}\text{\_}\mathrm{i}.\mathrm{y}\text{\_}\mathrm{i}.\left(\mathrm{k}\left(\mathrm{x},\mathrm{x}\text{\_}\mathrm{i}\right)\right)+\mathrm{b}\right]\right).}$

Naive Bayes (NB): NB is a probabilistic classification algorithm that works on the principles of Bayes’ theorem. It is based on the assumption of feature independence, meaning that each feature in the dataset is considered unrelated to others, given the class label  (Rahim et al., 2021; Hernández-Molinos et al., 2023). Naive Bayes efficiently categorizes instances by calculating the likelihood of each class given the observed features (Tua & Danar Sunindyo, 2019). Its simplicity, efficiency, and ability to handle diverse feature sets make it valuable in our framework. The mathematical form of Gaussian Naive Bayes can be simplified as follows: (5)${\displaystyle \mathrm{P}\left(\mathrm{y}|\mathrm{x}\right)=\left(1/\left(\text{sqrt}\left(2\mathrm{\ast }\pi \mathrm{\ast }{\sigma }^2\right)\right)\right)\mathrm{\ast }\mathrm{e}\stackrel{ˆ}{}\left(-\left({\left(\mathrm{x}-\mu \right)}^2/\left(2\mathrm{\ast }{\sigma }^2\right)\right)\right).}$

where P(yjx) represents the probability of class y given the feature vector x, μ is the mean (average) of the feature values for class y, σ² is the variance (spread) of the feature values for classy, e is the base of the natural logarithm, approximately equal to 2.71828.

Machine learning classifiers consist of parameters that can be adjusted to find optimized values. A systematic tuning approach was adopted to determine the optimal hyperparameters for each classifier. Initially, a hit-and-trial method was employed to explore a broad range of hyperparameter values for each classifier. Subsequently, an iterative tuning process was conducted, refining the hyperparameter values based on the performance metrics observed during multiple trial runs. RF showed significant results when the max depth parameter was set to “10”, and the n_estimators parameter was set to “500”. SVM showed the optimal performance when the kernel parameter was chosen as “poly”, and the Regularization Parameter C was set to “2”. NB achieved the highest performance when “GaussianNB” was implemented. The process of classifier tuning is shown in Fig. 5. The combination of RF, ‘, and NB was deliberately chosen due to their diverse and heterogeneous nature and complementary strengths. Each classifier contributes its unique characteristics: RF is based on decision trees; SVM employs a probability-based approach; and NB is rooted in Bayes’ theorem  (Ibrahim, Ghnemat & Hudaib, 2017; Tua & Danar Sunindyo, 2019; Azzeh et al., 2023).

Performance evaluation

After implementing the proposed framework, the most critical task is to examine its performance. Various machine learning tools provide several evaluation measures that comprehensively present performance, such as recall, precision, and G-mean (Iqbal, 2019). The best approach to evaluate the framework’s efficiency is to organize the results using a confusion matrix (Alsawalqah et al., 2020). To evaluate the effectiveness of the proposed approach, five widely recognized evaluation measures, namely precision, recall, accuracy, F-measure, and AUC, were thoughtfully applied (Balogun et al., 2019a).

Precision measures the accuracy of optimistic predictions, expressing the ratio of true positives to the total predicted positives. It is particularly relevant when minimizing false positives, which is crucial. Recall evaluates the model’s ability to capture all relevant instances. It is calculated as the ratio of true positives to actual positives. Accuracy is a fundamental metric representing the overall correctness of the model. It is calculated as the ratio of correctly predicted instances to the total instances, providing a holistic view of the model’s performance (Balogun et al., 2019a). The F-measure is the harmonic mean of precision and Recall. It offers a balanced assessment by considering both false positives and false negatives. AUC assesses the model’s ability to distinguish between positive and negative instances across various threshold levels. A higher AUC indicates superior discrimination capability, commonly used in binary and multiclass classification tasks (Lear et al., 2021). These measures were derived using a confusion matrix and were calculated with Python functions. The formula used to calculate each performance measure is given as:

(6)${\displaystyle \text{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$ (7)${\displaystyle \text{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$ (8)${\displaystyle \text{Accuracy}=\frac{\left(\mathrm{TP}+\mathrm{TN}\right)}{\left(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}\right)}}$ (9)${\displaystyle F-\text{measure}=\frac{2\ast \mathrm{Recall}\ast \mathrm{Precision}}{\mathrm{Recall}+\mathrm{Precision}}}$ (10)${\displaystyle \text{AUC}=\frac{1+\mathrm{TPr}-\mathrm{FPr}}{2}.}$

In the provided equations, TP corresponds to the accurate identification of defective software modules by the model, while TN signifies the correct recognition of non-defective software modules. The values FP and FN indicate discrepancies between the actual and predicted outcomes. Specifically, FP signifies instances where a non-defective module was predicted as defective. In contrast, FN indicates cases where a module, actually defective, was predicted as non-defective by the model. In contrast, TPr represents the actual positive rate, measuring the proportion of correctly predicted positive instances among all actual positives. FPr represents the false positive rate, indicating the proportion of incorrectly predicted negative instances among all negatives.

Genetic algorithm-based performance comparison

A comparative analysis has been conducted to evaluate the IECGA framework’s accuracy with and without feature selection (FS). The evaluation encompasses training and testing datasets, highlighting the influence of the GA-based feature selection on the overall effectiveness of the framework. Table 12 presents the performance comparison on the training datasets, showing the impact of FS using genetic algorithms within the IECGA framework. Figure 16 shows the GA-based accuracy comparison on training datasets.

In the case of the CM1, JM1, and PC4 datasets, the FS led to a significant increase in accuracy, from 91.67% to 92.11%, 81.31% to 81.83%, and 88.64% to 91.23%, respectively. This reflects the effectiveness of the GA-based FS in identifying and retaining the most informative features, potentially reducing computational costs. For MC2, MW1, PC1, and PC3 datasets, there is a marginal decrease in accuracy with FS from 74.42% to 73.26%, 95.43% to 94.86%, 94.95% to 94.74%, and 98.78% to 93.89%. While there is a slight decrease in accuracy, the reduced number of features contributes to significant savings in computational resources, including time, money, and manpower. This emphasizes the trade-off between feature richness and the practical efficiency of the classifiers in real-world applications. Figure 17 shows the graphical representation of the GA-based accuracy comparison on training datasets.

Table 13 presents the performance comparison on the testing datasets, showing the impact of FS using genetic algorithms within the IECGA framework.

In the MC2, PC1, and PC4 datasets, FS contributes to an accuracy boost from 68.42% to 71.05%, 93.63% to 95.1%, and 87.66% to 89.76%, respectively. The results indicate that the GA effectively selects and retains pertinent features, optimizing the predictive performance of the IECGA framework. Despite the accuracy of 85.86% in CM1 and 89.33% in MW1 datasets, the GA-based feature selection is influential in reducing computational resources. The maintained accuracy indicates that the GA accurately identifies and preserves the most relevant features. The marginal reduction in accuracy is observed in the JM1 and PC3 datasets, from 79.84% to 79.66% and 88.92% to 88.29%, respectively. The trade-off between accuracy and resource optimization is evident, emphasizing FS’s strategic application in scenarios where resource efficiency’s benefits outweigh the marginal decrease in accuracy. Figure 17 shows the graphical representation of the GA-based accuracy comparison on testing datasets.

While GA-based FS aims to retain the most relevant features, its intrinsic evolutionary nature may occasionally lead to the omission of some features that, although less impactful individually, contribute collectively to the framework’s predictive ability (Hamdia, Zhuang & Rabczuk, 2021; Katoch, Chauhan & Kumar, 2021). This accuracy drop should not be viewed solely as a compromise; instead, it reflects a strategic decision to prioritize computational efficiency and interpretability by discarding features that might introduce noise or redundancy.

Computational efficiency evaluation

Investigating the execution time of the framework with and without feature selection reveals notable improvements in computational efficiency. The application of FS consistently resulted in reduced execution time in both training and testing phases across all datasets. A comparative analysis of the execution time on both training and testing datasets has been presented in Table 14. Specifically, the time spent training with the original features significantly decreases when FS is employed, presenting a streamlined computational process. Similarly, in the testing phase, FS demonstrates its effectiveness by substantially lowering execution time, indicative of its role in optimizing the framework’s overall computational efficiency. Notably, the IECGA framework demonstrates enhanced efficiency, achieving a substantial average reduction of 51.52% in training times and 52.31% in testing times. These findings emphasize the practical advantages of incorporating FS, showcasing its potential to enhance computational performance.

A graphical representation of the comparative analysis of the execution time on both training and testing datasets has been reflected in Fig. 18.

Performance comparison

In this section, the accuracy of the proposed feature selection-based ensemble software defect prediction (IECGA) framework is compared to state-of-the-art software defect prediction techniques, as implemented in recent research conducted over the past five years. The comparison involves twenty published studies, with a focus on various datasets. Specifically, CM1 is the subject of ten studies; eight researchers examined JM1, MC2 was investigated in ten studies, MW1 was explored in eight studies, and PC1 was experimented with in eighteen studies. Additionally, the PC3 was analyzed in eight studies, and the PC4 was examined in seven. The results highlight the significance of incorporating heterogeneous base classifiers into feature selection-based ensemble classification, as it substantially enhances the accuracy of the software defect prediction process. The accuracy comparison between the IECGA framework and modern techniques is detailed in Table 15.

The graphical representation of the accuracy comparison of the IECGA framework with that of state-of-the-art techniques is shown in Fig. 19.