New hybrid multi-objective feature selection: Boruta-XGBoost

Dataset

In this section, the authors begin by introducing the dataset used in the study. The steps involved in the processes described in this article are then explained in a sequential manner to facilitate the readers’ understanding. The dataset (Johnson, 1996) used in this article focuses on predicting body fat percentage using anatomical measurements. It consists of data from 252 individuals, with 13 physical characteristics measured for each person. The specific names of these features are listed in Table 2.

General information about dataset

In this section, our aim is to obtain general information about the dataset. To achieve this, we perform several calculations for each feature, including count, mean, standard deviation, minimum, maximum, and quartile values (0%, 5%, 50%, 95%, 99%, 100%). These calculations provide us with an overview of the dataset and its characteristics.

As observed in Table 3 and Fig. 2, the features Weight, Height, Nip, and Ankle exhibit outlier values, resulting in a significant difference between the 99% and 100% quartile values. This discrepancy has an impact on our model and hinders the attainment of desirable results. In the upcoming section, we will address this issue and discuss methods to handle outlier data effectively.

In the subsequent step, our aim is to assess the strength of the relationship between variables in the dataset, a measure known as correlation. Through the analysis of the correlation matrix, we can determine which features exhibit a high degree of similarity.

To better understand the data distribution, we first visualized the target variable. Figure 3 presents a histogram of the body fat percentage, which shows a roughly unimodal and approximately symmetric distribution.

The presence of these potential outliers is visualized in Fig. 4, which presents box plots for each of the predictor variables. As illustrated, features such as Weight, Height, and Ankle exhibit data points beyond the typical range (1.5 times the interquartile range).

As depicted in Fig. 5, certain variables display a stronger positive correlation with each other. For instance, the Hip variable demonstrates this relationship. To enhance the speed and efficiency of our model, and to understand multicollinearity, we note variables that exhibit stronger positive correlations, although the Boruta method is generally robust to multicollinearity.

In the final step, we employ the joint plot method to gain a deeper understanding of the relationship between variables, as well as the individual distribution of each variable. This method facilitates a clearer comprehension of the association between independent and dependent variables, aiding us in selecting more significant features.

As depicted in Fig. 6, the joint diagram is comprised of three distinct sections. The central portion presents insights into the relationship between the independent and dependent variables, providing information regarding the joint distribution. The remaining two sections represent the marginal distributions for the dependent and independent variable axes.

Data pre-processing

Data mining refers to the process of extracting valuable patterns and models from a dataset. However, the quality of the data is crucial in uncovering these patterns. Raw and unprocessed data often contain empty, noisy, incomplete, inconsistent, and outlier values. Therefore, effective data pre-processing is necessary before conducting data mining to extract useful models from the dataset. Data pre-processing, which involves preparing the data before extracting information and models, plays a vital role in the accuracy and effectiveness of the resulting models (Zhang, Zhang & Yang, 2003). In this article, we will begin by performing the data pre-processing stage prior to any model or information extraction. Figure 7, illustrates the sequential workflow used to prepare the dataset for analysis. The stages include:

• a.

  Data cleaning: Handling inconsistencies such as missing values and outliers to ensure data quality.

• b.

  Data reduction: Employing feature selection techniques (in our case, Boruta) to reduce the dimensionality of the dataset by identifying the most relevant variables.

• c.

  Data transformation: Converting data into a suitable format for modeling, although this step was minimal in our study as features were primarily numerical.

• d.

  Data scaling: Normalizing the range of feature values (e.g., using Min-Max scaling) to ensure that all features contribute equally to model training, particularly for distance-based algorithms.

Data reduction

Data reduction refers to the process of reducing certain aspects of data. It can involve reducing the data’s count or dimensions, resulting in a smaller data volume. The goal of data reduction can vary, ranging from removing invalid data to generating statistical and summary data for various applications (García, Luengo & Herrera, 2015).

Data reduction is commonly carried out using row and column methods. The row method focuses on reducing the data sample through techniques like random and stratified sampling (Fan et al., 2015). On the other hand, the column method aims to reduce the number of variables in the dataset. There are typically three approaches for the column method. The first involves using domain knowledge for variable selection. The second approach is feature selection, which identifies the most important features (Liu & Motoda, 1998, 2007). The third approach is feature extraction, which creates new features from existing ones (Fan et al., 2021; Liu & Motoda, 1998). In this article, we will focus on the feature selection method since our primary objective is to intelligently select important and useful features from the body fat dataset using the Boruta algorithm, followed by prediction using XGBoost.

Feature selection is a crucial step in data mining that focuses on selecting relevant features from a dataset. It plays a role in separating relevant features from irrelevant ones, especially in high-dimensional data (Ramchandran & Sangaiah, 2018). Additionally, feature selection serves as a dimensionality reduction technique, enhancing classification accuracy by removing irrelevant or unnecessary features (Dey et al., 2018). The feature selection process involves five fundamental steps, As you can see in Fig. 8, and the choices made at each step directly impact the performance of feature selection (Langley, 1994). In the following section, we will briefly outline these steps.

Step 1 involves determining the search direction and starting point for feature selection. The search can begin with an empty set and gradually add features in each iteration, known as forward searching. Alternatively, the search can start with a full set and eliminate features iteratively, known as reverse elimination search. A two-way search strategy involves simultaneously adding and removing features. Another option is to randomly select features to create an initial subset (Ang et al., 2015). Note that Boruta uses an iterative approach based on feature importance comparison.

Step 2 involves determining the search strategy for feature selection. An effective search strategy should include global search capabilities, fast convergence to the optimal solution, efficient local search, and computational efficiency (Gheyas & Smith, 2010). Search strategies can be categorized into three groups: exponential, sequential, and random, as studied by various researchers (Chan et al., 2010; Kumar & Minz, 2014; Muni, Pal & Das, 2006; Ruiz, Riquelme & Aguilar-Ruiz, 2005).

Step 3 involves determining the evaluation criteria for assessing the effectiveness of different feature selection techniques. Evaluation criteria are essential for selecting the most appropriate features. The choice of evaluation criteria depends on the specific problem and data being analyzed. Commonly used evaluation criteria in feature selection include Filter (Chandrashekar & Sahin, 2014), Wrapper (Kohavi & John, 1997), Embedded (Zheng & Casari, 2018), and Ensemble (Opitz & Maclin, 1999). Boruta is a Wrapper method, using Random Forest as the embedded evaluation model.

Step 4 involves determining the stop criteria for the feature selection process. The stop criteria determine when the process will terminate. An appropriate stop criterion helps avoid overfitting and reduces computational complexity, leading to a more efficient generation of the optimal feature subset. The choice of stop criteria may be influenced by decisions made in earlier steps. Common features of stopping criteria are given below (Ang et al., 2015; Kumar & Minz, 2014; Liu & Yu, 2005):

• A predefined number of features is reached.

• A predefined number of iterations is reached. (A common stopping criterion for Boruta, besides stability).

• The performance improvement between iterations falls below a threshold.

• An optimal feature subset is deemed to be found based on some evaluation function or stability criterion. (Boruta stops when all features are confirmed or rejected, or max iterations are hit).

Step 5 involves validating the final results obtained from the feature selection process. One way to validate the results is by comparing them with prior knowledge of the data, if available. This can be done by comparing the selected features with a known set of relevant features (Liu & Motoda, 1998; Zeng et al., 2009). However, in real-world applications, such prior knowledge is often not available. In such cases, indirect validation methods are used, such as cross-validation, confusion matrix, Jaccard similarity-based measure, and random index. Cross-validation is the most commonly used validation technique (Bolón-Canedo, Sánchez-Maroño & Alonso-Betanzos, 2013).

Feature extraction is a method that goes beyond selecting important features and aims to create new features by combining dataset variables in linear or non-linear ways (Liu & Motoda, 2013). However, in this article, our focus is on selecting useful features rather than creating new ones. Therefore, we did not perform any feature extraction on the variables of the body fat dataset.

XGBoost

XGBoost is a powerful and widely adopted machine learning algorithm based on the gradient boosting framework (Chen & Guestrin, 2016). It has gained significant popularity due to its high performance and computational efficiency, particularly in machine learning competitions and industrial applications. XGBoost sequentially trains weak learners (typically decision trees), where each new model attempts to correct the errors made by the previous ensemble. Key features of XGBoost include:

• Regularization: XGBoost incorporates L1 (Lasso) and L2 (Ridge) regularization into its objective function to prevent overfitting and produce simpler models.

• Handling missing values: XGBoost can intrinsically handle missing values within the data.

• Built-in Cross-Validation: The algorithm provides the capability to perform cross-validation at each boosting iteration.

• Parallel processing: XGBoost is designed for efficient utilization of multi-core processors and can perform tree construction in parallel.

• Tree pruning: It employs both “max depth” constraints and pruning based on negative loss reduction (gamma parameter).

The overall objective function that XGBoost aims to minimize combines the loss function and a regularization term:

$$\mathbf{O}\mathbf{b}\mathbf{j}\left(\text{θ}\right)=\sum _{\mathbf{i}=1}^{\mathbf{n}}\mathbf{l}\left({\mathbf{y}}_{\mathbf{i}},{\hat{\mathbf{y}}}_{\mathbf{i}}\right)+\sum _{\mathbf{k}=1}^{\mathbf{K}}\text{Ω}\left({\mathbf{f}}_{\mathbf{k}}\right)$$where:

• $\mathbf{l}\left({\mathbf{y}}_{\mathbf{i}},{\hat{\mathbf{y}}}_{\mathbf{i}}\right)$ is the loss function, a differentiable convex function that measures the difference between the true target value ${\mathbf{y}}_{\mathbf{i}}$ and the prediction ${\hat{\mathbf{y}}}_{\mathbf{i}}$ for the i-th training sample.

• ${\sum }_{\mathbf{i}=1}^{\mathbf{n}}$ denotes the summation of the loss over all n training samples.

• $\text{Ω}\left({\mathbf{f}}_{\mathbf{k}}\right)$ is the regularization term for the k-th tree. This term penalizes the complexity of the model to prevent overfitting. It is typically defined as $\text{Ω}\left({\mathbf{f}}_{\mathbf{k}}\right)=\text{γ}\mathbf{T}+\frac{1}{2}\lambda \parallel \mathbf{w}{\parallel }^2$, where T is the number of leaves, w is the vector of leaf weights, and $\text{γ}$ and $\lambda$ are regularization parameters.

• ${\sum }_{\mathbf{k}=1}^{\mathbf{K}}$ denotes the summation of the complexity penalty over all K trees in the ensemble.

• ${\mathbf{f}}_{\mathbf{k}}$ represents the k-th decision tree in the model.

• $\text{θ}$ represents the set of all parameters to be learned.

Optimization

Optimization is the process of finding the most optimal solution or outcome for a problem by minimizing effort or maximizing gain. It involves determining the conditions that yield the maximum or minimum value of a specific function. By employing mathematical evaluation and modeling the problem as an objective function, optimization aims to search for the best solution among all possible alternatives. This mathematical technique is widely used to solve problems and make informed decisions in various fields (Burke & Kendall, 2005).

As depicted in the following Equation, the standard representation of an optimization problem involves minimizing the objective function (Sengupta, Gupta & Dutta, 2016):

$\mathrm{m}\mathrm{i}{\mathrm{n}}_{x}f\left(x\right)$ subject to:

$$g_i\left(x\right)\le 0,i=1,\dots ,m$$

$$h_j\left(x\right)=0,j=1,\dots ,p$$where $f\left(x\right)$ is the function to be optimized, $g_i\left(x\right)$ represents $m$ inequality constraints, and $h_j\left(x\right)$ represents $p$ equality constraints. In wrapper feature selection like Boruta, the objective function is implicitly the performance of the underlying machine learning model, evaluated iteratively.

Advanced feature selection algorithms

Throughout history, problem-solving approaches have often been based on intuition or heuristics. This reliance has contributed to the success and popularity of meta-heuristic algorithms among researchers, which mimic natural processes to find acceptable solutions to complex problems through trial and error (Yang, 2010). Beyond meta-heuristics, other advanced techniques like wrapper methods have been developed for feature selection.

Advanced feature selection algorithms, including meta-heuristics and wrapper methods like Boruta, are designed to tackle complex optimization problems that cannot be effectively solved using classical or simpler filter methods. These algorithms are considered highly effective and widely applicable, especially for real-world problems (Dokeroglu, Deniz & Kiziloz, 2022). The main advantage lies in their effectiveness and general applicability. Unlike classical optimization algorithms, these methods focus on the exploration and exploitation of the feature space, often integrating machine learning models directly into the evaluation process. While parameter tuning remains important, they aim to reduce the time and cost associated with finding improved feature subsets compared to exhaustive search (Ólafsson, 2006). The general framework for many search-based algorithms (especially meta-heuristics) is as follows (Almufti, 2019):

• (1)

  Initialization: Initialize a set of candidate solutions (feature subsets) randomly or using a heuristic.

• (2)

  Evaluation: Evaluate the fitness of each candidate solution using an objective function (often involving a learning model).

• (3)

  Iteration: Repeat the following steps until a stopping criterion is met: a. Selection: Select a subset of candidate solutions based on their fitness values. b. Variation: Apply rules or operators to the selected candidates to generate new candidate solutions. c. Evaluation: Evaluate the fitness of the new candidate solutions. d. Update: Update the set of candidate solutions based on the fitness values.

• (4)

  Termination: Stop the algorithm when a stopping criterion is met.

Boruta follows an iterative process based on comparing real vs shadow feature importance.

The Boruta algorithm

Boruta is a wrapper feature selection algorithm designed to identify all statistically relevant features in a dataset concerning a target variable (Kursa & Rudnicki, 2010). The name ‘Boruta’ is derived from a demon in Slavic mythology who dwelled in pine forests; the authors of the algorithm chose this name metaphorically for its function of exploring the forest of features. It is built upon the Random Forest algorithm and utilizes a clever approach to determine feature importance. The core idea of Boruta is to compare the importance of the original features against the importance of randomized, “shadow” features. The main steps of the Boruta algorithm are as follows:

• (1)

  Create shadow features: For each original feature in the dataset, a copy is created, and its values are randomly permuted across the samples. These new features are termed “shadow features.”

• (2)

  Train model: A Random Forest model is trained on the extended dataset containing both original and shadow features.

• (3)

  Calculate importance: Feature importance scores (typically Mean Decrease Accuracy or Mean Decrease Gini) are computed for all features (original and shadow) using the trained Random Forest.

• (4)

  Determine threshold: The maximum importance score among all shadow features (Maximum Z-score Among Shadow Attributes—MZSA) is identified and serves as the importance threshold for that iteration.

• (5)

  Compare and decide: For each original feature, its importance score is compared against the MZSA. A statistical test (usually a binomial test across multiple iterations) is used to assess whether a feature’s importance is significantly higher than the threshold. Based on this comparison, features are classified into three categories:

  • Confirmed: Features whose importance is consistently and significantly higher than the MZSA.

  • Rejected: Features whose importance is consistently and significantly lower than or equal to the MZSA.

  • Tentative: Features whose importance scores are close to the MZSA, making a definitive decision difficult within the current iterations.

• (6)

  Iterate: Rejected features and shadow features are removed from the dataset, and steps 2–5 are repeated until all features are either Confirmed or Rejected, or a predefined maximum number of iterations is reached. Any remaining Tentative features might be resolved by comparing their median importance against the median importance of the shadow features across all runs.

The final output of Boruta is the set of “Confirmed” features, considered relevant to the target variable. This method is known for its ability to identify all relevant features, including those that might be weakly correlated or interact with others.

Feature selection with advanced algorithms

Feature selection is a complex problem, prompting researchers to utilize advanced artificial intelligence and statistical methods. For years, efforts have focused on finding optimal ways to enhance accuracy and efficiency in this scientific domain. Meta-heuristic algorithms and wrapper methods like Boruta are recognized for being more effective than simpler methods in improving feature selection results. Recent studies demonstrate the ability of these algorithms to yield high-accuracy results and refine the feature selection process (Yusta, 2009).

Here are the general steps to perform feature selection using advanced wrapper algorithms like Boruta (followed by prediction with XGBoost):

• (1)

  Define the problem: The objective is to predict body fat percentage using an optimal subset of anatomical features.

• (2)

  Choose the feature selection algorithm: The Boruta algorithm is selected for its capability to find all relevant features.

• (3)

  Execute Boruta: The Boruta algorithm is run on the training dataset. It iteratively compares the importance of real features against shadow features to identify ‘Confirmed’ features.

• (4)

  Select the prediction model: The XGBoost algorithm is chosen as the final prediction model due to its strong performance.

• (5)

  Train the final model: An XGBoost model is trained using only the features identified as ‘Confirmed’ by Boruta on the training data.

• (6)

  Evaluate performance: The performance of the trained XGBoost model is assessed on a separate test dataset using appropriate metrics (e.g., RMSE, R2).

• (7)

  Validate: Techniques like cross-validation (potentially used during XGBoost parameter tuning or for overall evaluation) ensure the robustness and generalizability of the results.

The proposed Bruta-XGBoost hybrid framework

Our proposed method for intelligent feature selection and high-accuracy prediction is a two-stage hybrid framework that sequentially combines the Boruta algorithm and Extreme Gradient Boosting (XGBoost). The core principle of this framework is to first purify the feature space using a statistically robust method before engaging a powerful predictive model. This strategic separation allows each component to perform its specialized task optimally. Figure 9 illustrates the workflow of our framework.

The stages of the framework are detailed below:

Stage 1: Feature subset identification with Boruta

The first stage is dedicated to identifying all features that are statistically relevant to the target variable (body fat percentage). For this, we employ the Boruta algorithm. Unlike greedy algorithms that might only select a small set of highly predictive features, Boruta aims to find all features that carry significant information. Its process, as described in ‘The Boruta Algorithm’, involves creating ‘shadow’ features and iteratively comparing the importance of original features against these random-shuffled copies within a Random Forest model. The output of this stage is a subset of features labeled as ‘Confirmed’. This ‘Confirmed’ set represents a clean and reliable feature space, free from irrelevant and noisy variables, which forms the input for the next stage.

Stage 2: High-performance prediction with XGBoost

In the second stage, we use the ‘Confirmed’ feature subset produced by Boruta to train a prediction model. We selected XGBoost for this task due to its well-documented high performance, computational efficiency, and built-in regularization mechanisms that protect against overfitting. The XGBoost model is trained only on the pre-selected, relevant features. This focused approach allows the model to learn the underlying patterns more effectively, without being distracted by irrelevant data, leading to a more accurate and generalizable prediction.

To provide a clear algorithmic representation, the pseudocode for our proposed framework is presented below:

Stage 1: Feature Selection with Boruta

 1. Print: "Stage 1: Identifying relevant features…"
  2. Run Boruta algorithm on TrainingData using TargetVariable
    ○ max_iterations = 100
    ○ significance_level = 0.01
  3. Store selected features in ConfirmedFeatures
  4. Print number of features identified
  5. Prepare datasets using only selected features:
    ○ TrainingData_Selected = columns of TrainingData corresponding to ConfirmedFeatures
    ○ TestData_Selected = columns of TestData corresponding to ConfirmedFeatures
    ○ TrainingTarget = TrainingData [TargetVariable]
    ○ TestTarget = TestData [TargetVariable]

Stage 2: Train XGBoost model

1. Print: "Stage 2: Training XGBoost model…"
2. Train XGBoost model:
    ○  XGB_Model = TrainXGBoost(TrainingData_Selected, TrainingTarget, hyperparameters = Tuned_XGB_Params)
3. Print: "Making predictions on test data…"
4. Predictions = XGB_Model.predict(TestData_Selected)
5. Print: "Evaluating model performance…"
6. PerformanceMetrics = Evaluate(Predictions, TestTarget)
7. Return PerformanceMetrics

Computational complexity analysis

• Stage 1 (Boruta): The complexity of Boruta is primarily driven by the Random Forest models it trains iteratively. It is approximately O(I × T × M logM × N′), where I is the number of Boruta iterations, T is the number of trees in the forest, M is the number of samples, and N’ is the number of features considered at each split.

• Stage 2 (XGBoost): The training complexity of XGBoost is roughly O(K × D × M logM), where K is the number of boosting rounds (trees) and D is the maximum tree depth. In our framework, this is applied to a reduced feature set N_sel.

While the upfront cost of Boruta can be significant, this investment leads to a substantial reduction in the dimensionality from N to N_sel. This not only simplifies the final model but can also lead to a faster training phase for the subsequent XGBoost model, especially when N is very large. The main benefit, however, is not just computational, but the improved accuracy and robustness of the final model.

The results of this analysis are summarized in Table 4, and a visual comparison of the total execution times is presented in Fig. 10.

Model evaluation

Performance metrics, also known as error measures, play a crucial role in evaluation frameworks across different fields (Botchkarev, 2018). The process of evaluation involves collecting, analyzing, and interpreting data to assess the performance of a system. By doing so, we can determine the system’s effectiveness and efficiency based on the information gathered. Evaluation is an essential step in understanding how well a system performs and helps us make informed decisions and improvements (Thorpe, 1988).

In this article, we assess the model’s performance using four key criteria: Mean Square Error (MSE) (Das, Jiang & Rao, 2004), Root Mean Square Error (RMSE) (Willmott & Matsuura, 2005), Mean Absolute Error (MAE) (Chai & Draxler, 2014), and R-Squared (R²) (Figueiredo Filho, Júnior & Rocha, 2011). Detailed explanations and calculation methods for each measure can be found in Table 5.

Experimental setup

In selecting the baseline models for comparison, we aimed to cover a wide spectrum of established machine learning techniques, including both classic linear models and powerful non-linear ensembles. While some of these algorithms have been foundational for years, our suite includes modern, high-performance methods such as XGBoost, LightGBM, and CatBoost. These gradient boosting decision tree (GBDT) based algorithms, in particular, have consistently demonstrated state-of-the-art performance on tabular data problems. As confirmed by numerous recent studies and large-scale machine learning competitions, they remain the dominant and often superior choice for structured data, frequently outperforming more recent deep learning architectures in terms of both predictive accuracy and computational efficiency (Grinsztajn, Oyallon & Varoquaux, 2022; Shwartz-Ziv & Armon, 2022). Therefore, benchmarking our proposed framework against these powerful algorithms provides a rigorous and contemporary test of its efficacy. The dataset was partitioned into a training set (70% of the data) and a testing set (30%). A 10-fold cross-validation scheme was employed on the training set to ensure the robustness and generalizability of our findings.

For the 16 baseline models evaluated in ‘Computational Performance Analysis’, we utilized the default hyperparameter settings provided by the scikit-learn library. This approach was chosen to establish a standardized baseline performance, representing how these models would perform “out-of-the-box” without extensive tuning.

For our proposed Boruta-XGBoost framework, the XGBoost component was subjected to hyperparameter tuning to optimize its performance. We used a Grid Search with 5-fold cross-validation on the training data to find the best combination of key parameters, including n_estimators, max_depth, and learning_rate. The Boruta algorithm was run with its default parameters, which are generally robust. This ensures that we are evaluating the peak performance of our proposed method against the standard baseline performance of others. All experiments were conducted in a Python 3.x environment using libraries such as Scikit-learn, XGBoost, and BorutaPy.

Baseline performance: body fat prediction using all features

In this section, we apply 16 machine-learning algorithms to the body fat dataset. Among these 16 machine learning algorithms, six are linear algorithms and 10 are non-linear algorithms.

The Boruta-XGBoost method achieved an R² of 0.88 by utilizing only 6 out of the 13 original features. To provide insight into this selection process, Fig. 11 visualizes the importance scores calculated by the Boruta algorithm for all features. The plot clearly distinguishes between the ‘Confirmed’ features, which were retained for model training, and the ‘Rejected’ features. It is evident that features such as ‘Abdomen’ and ‘Weight’ were identified as highly significant, while others like ‘Ankle’ and ‘Knee’ were deemed irrelevant, justifying their exclusion and leading to a more parsimonious model.

The high predictive power of our framework is further illustrated in Fig. 12. This plot compares the body fat percentages predicted by the Boruta-XGBoost model against the actual values from the test set. The data points cluster tightly around the ideal y = x line, indicating a strong correlation and low error. The resulting R² value of 0.88 visually confirms that the model can explain the vast majority of the variance in the data, making it a highly reliable predictive tool.

At first, we use linear algorithms of Multiple Linear Regression (MLR), Partial Least Squares Regression (PLS) (Wold, Sjöström & Eriksson, 2001), Ridge Regression (Hoerl & Kennard, 1970), Lasso Regression (Tibshirani, 1996), Elastic-Net Regression (Zou & Hastie, 2005), and Support Vector Regression (SVR) (Drucker et al., 1996) to predict body fat, and you can see the results obtained from these linear algorithms in Table 6.

As you can see in Fig. 13, among the linear algorithms, the multiple linear regression algorithm with an accuracy of 0.763 has the highest prediction rate in the body fat dataset.

In the next step, we use the non-linear algorithms of K-Nearest Neighbors (KNN) (Fix & Hodges, 1989), Non-Linear Support Vector Regression (N-SVR) (Cortes & Vapnik, 1995), Artificial Neural Network (ANN) (McCulloch & Pitts, 1943), Classification and Regression Trees (CART) (Breiman et al., 2017), Bagged Trees (Breiman, 1996), Random Forests (RF) (Breiman, 1996), Gradient Boosting Machines (GBM) (Friedman, 2001), XGBoost (Chen & Guestrin, 2016), LightGBM (Ke et al., 2017), and Category Boosting (CatBoost) (Prokhorenkova et al., 2018) to predict body fat, and the results obtained from these non-linear algorithms can be seen in Table 7.

As you can see in Fig. 14, among the nonlinear algorithms, the nonlinear support vector regression algorithm with an accuracy of 0.77 has the highest prediction rate in the body fat dataset.

Computational performance analysis

In addition to predictive accuracy, model efficiency is a critical factor, especially as datasets grow in size and complexity. To evaluate the computational performance of our proposed framework, we conducted an empirical analysis of the training times. We measured the wall-clock time required to train our Boruta-XGBoost framework and compared it against the training times of key baseline models: XGBoost (on the full feature set), the best-performing baseline (Non-Linear SVR), and another powerful ensemble (Random Forest). The experiments were run on a consistent hardware environment Intel Core i7 CPU @ 2.8 GHz with 16 GB RAM.

As we mentioned in ‘Computational Complexity Analysis’, the Boruta algorithm introduces a significant upfront computational cost (2.50 s) for feature selection. This is expected, as Boruta iteratively trains multiple Random Forest models. However, once the feature set was reduced from 13 to 6, the training time for the subsequent XGBoost model (0.20 s) was more than twice as fast as training XGBoost on the full feature set (0.45 s).

While the total execution time of our framework is higher than the baseline models for this relatively small dataset, the analysis highlights a crucial trade-off. The investment in robust feature selection, though computationally intensive, yields a more accurate and parsimonious final model. For datasets with a much larger number of initial features (N), the reduction to a small subset (N_sel) is expected to result in even more substantial savings in the model training phase, potentially making the overall pipeline more efficient. This analysis underscores that the primary benefit of our framework is the significant boost in predictive accuracy, achieved through a systematic, albeit computationally intensive, feature selection process.

Proposed method performance: prediction using Boruta-selected features

As you can see in Table 8 and Fig. 15, the accuracy (R²) on the test data using our proposed Boruta-XGBoost method is 0.88, which represents the highest accuracy among all algorithms evaluated in this article.

As you can see in Fig. 15, the accuracy of the test data in our proposed method is 0.88, which has the highest accuracy among the machine learning algorithms mentioned in this article. In summary, we seek the best cost value to select useful and important features by using an intelligent feature selection method that is a combination of Boruta-XGBoost. By examining Fig. 16, we find that the selection of six features, which is equivalent to 46% of the features of the entire body fat dataset, the model achieved a higher level of accuracy compared to the baseline algorithms that utilized all available features.

As visually summarized in Fig. 17, there is a notable variation in performance across the different modeling approaches. The observed decline in accuracy for certain methods can be attributed to several factors:

• (1)

  Linear vs non-linear relationships: The linear models, such as Multiple Linear Regression (MLR), show competent but limited performance (R² ≈ 0.76). This suggests that while linear relationships exist between the anatomical measurements and body fat percentage, they are insufficient to capture the full complexity of the underlying biological system. The problem inherently contains non-linear patterns that these models cannot effectively learn.

• (2)

  Sensitivity to irrelevant features: The performance of powerful non-linear models like Non-Linear SVR (R² = 0.77) and even XGBoost when applied to the full dataset is strong, but still suboptimal. This indicates that the presence of all 13 features, some of which are noisy or redundant (as identified by Boruta), likely hinders their learning process. These models may be assigning importance to irrelevant variables or struggling to distinguish signal from noise, thus limiting their predictive ceiling.

• (3)

  The synergy of feature selection and prediction: The superior performance of our Boruta-XGBoost framework (R² = 0.88) provides a clear explanation for this performance gap. By first using Boruta to perform a rigorous statistical cleaning of the feature space, we provide the XGBoost model with a dataset containing only highly relevant, information-rich features. This allows XGBoost to build a more focused, robust, and ultimately more accurate model. The decline in accuracy of other methods is therefore explained by their inability to either handle non-linearities (in the case of linear models) or effectively navigate a noisy feature space (in the case of non-linear models applied to the full dataset).