A new parametric information-gain criterion for tree-based machine learning algorithms

Introduction

Decision Trees (DTs) are still among the most widely used models in Machine Learning (ML), despite their roots being traced to as far back as the early 1960s (Morgan & Sonquist, 1963). This enduring relevance is owed to these models’ underlying simplicity and practical performance. Succinctly, DTs operate by recursively splitting the dataset into subsets based on feature values, aiming to create groups that are as pure as possible concerning a given target variable. Further aiding their adoption, the output of these recursive operations can be represented in a tree-like flowchart structure, greatly increasing their interpretability (Kotsiantis, 2013). Moreover, DTs also introduce a low computational cost and are compatible with both classification and regression tasks, which has justified their use in a wide range of applications, including, for example, in the medical, financial, and industrial sectors (Costa & Pedreira, 2023; Mienye & Jere, 2024).

Since their inception over half a century ago, DTs have lost their performance advantage compared to the best of the supervised learning approaches (James et al., 2021). However, they remain a critical foundational component of more powerful ensemble methods like Random Forests (RFs) and Gradient Boosted Trees, which are algorithms that remain highly competitive, particularly on tabular datasets, against even state-of-the-art neural network models (Grinsztajn, Oyallon & Varoquaux, 2022; Uddin & Lu, 2024). Moreover, there has been a newfound interest in the development of DTs due to their interpretability (Hwang, Yeo & Hong, 2020; Hernández et al., 2021), as the black-box nature commonly associated with most predictive and classifier models becomes an increasingly greater issue in an age where explainable machine learning is becoming a necessary condition (Rudin, 2019; Roscher et al., 2020).

Central to the induction (or construction) of DTs is the process of recursive data partitioning (Costa & Pedreira, 2023), which is guided by the output of a discrete function over the input attributes (Kotsiantis, 2013). The selection of the most appropriate function is typically determined by some splitting criteria, such as information gain, the Gini value, or the Gain ratio (GR), with the selection of these criteria directly influencing the structure and performance of the resulting tree. In the more (now) classical implementations of DT models, such as the Iterative Dichotomiser 3 (ID3) algorithm (Quinlan, 1986), information-theoretic measures (e.g., Shannon entropy) have well-established roles as splitting criteria. Nonetheless, past research, such as the work done by Nowozin (2012), shows that potential limitations remain in standard methods, as their performance is not uniform across different domains, data distributions, or learning objectives.

Recently, there has been additional research in either the development of novel splitting criteria (Leroux, Boussard & Dès, 2018; Ayllón-Gavilán et al., 2025; Hwang, Yeo & Hong, 2020; Loyola-González, Ramírez-Sáyago & Medina-Pérez, 2023) or in re-adapting existing methods with less common information-theoretic measures (Ignatenko, Surkov & Koltcov, 2024; Nowozin, 2012; Maszczyk & Duch, 2008; Gajowniczek, Zabkowski & Orłowski, 2015). In these works, the goal becomes the development of alternative and corrected measures aimed at improving split balance, reducing overfitting, and enhancing the generalization of the output DTs, with the adequate selection of the splitting criteria becoming particularly important in the context of high-dimensional, imbalanced, or noisy datasets. In a detailed analysis provided in Hernández et al. (2021), it is shown that no one split evaluation measure is capable of consistently outperforming all others. Alternatively, one approach is to combine multiple evaluation measures and select the candidate splits that better adapt to the input data (Loyola-González, Ramírez-Sáyago & Medina-Pérez, 2023).

However, in this article, we introduce an alternative perspective, based on the development of a more generic notion of entropy that is more parametric than existing approaches, and that enables the generalization of existing information-theoretic criteria most commonly used as the basis of splitting procedures. Through the added degrees of freedom provided by this approach, it is possible to construct splits that better adjust to input data. This method, evaluated for both open data repositories and synthetic datasets, is seen to achieve performance improvements of over 40% in F1-score relative to standard methods in common datasets such as the ‘Wine Quality’ dataset, when using RF algorithms. Moreover, we provide a Bayesian statistical analysis to assess that the proposed method presents a statistically significant improvement while never underperforming, with at worst, matching the performance of the best standard model.

The remainder of this article is organized as follows. ‘Decision Trees for Classification’, provides a generic discussion regarding DTs, their construction, and their splitting criteria. ‘Generalized Entropy’, introduces the 5-parameter expression for the generalization of entropy and its formulation into a target function for its use as a DT splitting criterion. ‘Description of Computational Experiments’, describes the datasets used for computational experiments, experimental methodology, and result evaluation metrics. ‘Results of Computational Experiments’, provides the summarized results obtained from the computational experiments and discusses the implementation of a Bayesian statistical analysis to validate the quality of the introduced method. ‘Discussion’, interprets the obtained results and statistical analysis. ‘Conclusions’, provides the concluding remarks regarding the introduced method. ‘Proofs of Known Entropies’, shows the mathematical proof of the generalizations possible to achieve with the presented parametric entropy expression. ‘Complete Results for Proposed Method’, provides additional tabular results.

Decision trees for classification

Multiple versions of the DT algorithm exist; however, most notable are perhaps their most “traditional” forms as seen in the case of the original ID3 (Quinlan, 1986), its successor C4.5 (Quinlan, 1993), and the Classification and Regression Tree (CART) (Breiman et al., 1984). In these forms, the structure of a DT consists of internal and terminal nodes. The former represent logical tests (splits) where each split has a binary outcome (true or false); while the latter are the leaves of the tree and correspond to output predictions (Costa & Pedreira, 2023). DTs in formulations such as CART are compatible with classification and regression tasks, hence leaves can correspond to either labels or constant numbers. These values are selected during the induction of the tree, where, starting at a root node that contains the full training dataset, this input space is recursively partitioned into homogeneous regions with respect to a given target variable. This forms the basis for the splitting process.

In the case of ID3, C4.5, and CART, they all take on greedy approaches to constructing DTs (Han, Kamber & Pei, 2012), meaning that they will evaluate all possible splits across all features and select the one that best separates the data according to a predefined criterion. The most commonly used criteria include information gain (based on information-theoretic measures), the Gini index, and the gain ratio. These measures assess the purity of the resulting subsets, aiming to maximize class purity in the child nodes, where a pure partition would mean that every element would have the same label.

Information gain (IG), used in ID3, is calculated as the reduction in entropy, e.g., using the Shannon entropy, before and after the split. Mathematically, this corresponds to computing (Mienye & Jere, 2024)

(1) $$IG(S,A)=H(S)-\sum _{v\in Values(A)}\frac{|S_v|}{|S|}H(S_v),$$where S is our dataset, A is an attribute of S, $Values(A)$ are the unique values in attribute A, $S_v$ the subset of S for which attribute A takes on value $v$, and H the entropy function, which in the case of Shannon’s entropy is given by Shannon (1948)

(2) $$H(S)=-\sum _{i=1}^n{P}_i{\log }_2{P}_i,$$where $n$ is the number of unique classes in the set S and $P_i$ is the proportion of the samples in the set that belong to class $i$. Essentially, the higher the information gain, the more adequate an attribute is in partitioning the data, as the subsets become more homogeneous. In more advanced DT algorithms, such as C4.5, Gain Ratio is used, a method where IG is normalized in order to correct its bias toward attributes with many distinct values (Quinlan, 1993; Han, Kamber & Pei, 2012). The gain ratio for a dataset S and an attribute A is given by

(3) $$GainRatio(A)=\frac{IG(A)}{SplitInfo(S,A)},$$where

(4) $$SplitInfo(S,A)=-\sum _{v\in Values(A)}\frac{|S_v|}{|S|}{\log }_2\left(\frac{|S_v|}{|S|}\right).$$

Lastly, the final of the most common splitting criteria is the Gini index (or value or impurity), employed by the CART algorithm. This metric measures the probability of misclassifying a randomly chosen instance from the dataset and is defined as Jost (2006) and Kotsiantis (2013)

(5) $$Gini(S)=1-\sum _{\mathit{i}=1}^{\mathit{n}}{\mathit{P}}_i^2.$$

Once the best split is identified, the dataset is divided accordingly, and the algorithm recurses on each subset. This process continues until a stopping criterion is met, such as reaching a maximum tree depth, achieving a minimum number of samples per node, or obtaining pure nodes (Nowozin, 2012). In classification tasks, each leaf node is assigned the majority class of the samples it contains.

Currently, DTs are seldom used in isolation, being much more commonly employed as a component of ensemble models. The most well-known of these is perhaps Random Forests (RFs), a method based on the induction of multiple DTs and then combining their outputs to improve predictive accuracy and control overfitting (Breiman, 2001). This algorithm operates by training each tree on a different bootstrap sample (random sample with replacement) of the dataset and using a random subset of features at each split to ensure diversity among trees. For classification, predictions are made by majority voting across trees; for regression, predictions are averaged. This combination of bagging (bootstrap aggregating) and random feature selection makes RFs more robust to noise, resistant to overfitting, and effective for high-dimensional data. A more comprehensive insight into these methods can be found in works such as Mienye & Jere (2024) and Costa & Pedreira (2023).

The focus of this article lies in the development of novel criteria for the splitting of tree-based algorithms and on surpassing some of the limitations that remain within the standard methods. In this sense, there have been previous attempts at this issue. For instance, in Leroux, Boussard & Dès (2018), a balanced gain ratio is discussed, aimed at addressing the bias towards unbalanced splits in the GR method used by Quinlan (1986), essentially by correcting the split information by a constant value. Empirically evaluated, this approach is seen to limit the depth of resulting trees with an improvement in classification accuracy. In other settings, splitting criteria are adapted to serve particular tasks. For instance, in Ayllón-Gavilán et al. (2025), a splitting criterion is defined for use with ordinal classification. Another example can be found in Hwang, Yeo & Hong (2020) where the main goal is not to achieve the best performance, but rather to lead to the creation of the most interpretable tree possible.

More in line with the motivation of this article is the work done by Nowozin (2012), where the classical entropy estimators, such as Shannon entropy or Gini index, are replaced, in this case, with the Grassberger entropy estimator, enabling an increase in predictive performance during classification tasks. We can further see the use of nonclassical entropies for the induction of DTs in Gajowniczek, Zabkowski & Orłowski (2015) and Maszczyk & Duch (2008). In both of these past examples, the authors employed Rényi and Tsallis parametric entropies. Introduced by Rényi (1961), the Rényi entropy of order $\alpha$ of a set S, and with $0<\alpha <\mathrm{\infty }$ and $\alpha \ne 1$, is defined as

(6) $$H_{\alpha }(S)=\frac{1}{1-\alpha }\log \left(\sum _{i=1}^n{P}_i^{\alpha }\right).$$

In case $\alpha \in \{0,1,\mathrm{\infty }\}$, it is defined as

(7) $$H_{\alpha }(S)=\underset{x\to \alpha }{lim}{H}_x(S).$$

Rényi’s entropy generalizes various other notions of entropy. For instance, as $\alpha \to 1$ (notation meaning: when $\alpha$ tends to $1$ by a valid branch), the Shannon entropy is recovered. Alternatively, the Tsallis entropy of a set S, introduced by Tsallis (1988), is defined as

(8) $$S_q(S)=k\frac{1}{q-1}(1-\sum _{\mathit{i}=1}^{\mathit{n}}{\mathit{P}}_{\mathit{i}}^{\mathit{q}}),$$where $k\in {\mathbb{R}}^{+}$ and $q\in \mathbb{R}$. Tsallis’ entropy is also capable of recovering other entropic definitions. For example, as $q\to 1$, the Boltzmann–Gibbs entropy is obtained.

Advancements are still occurring in this field, namely in the use of dynamically adjustable criteria at split time (Loyola-González, Ramírez-Sáyago & Medina-Pérez, 2023), or in regards to the use of deformed entropies for the improvement of target functions. One such recent example of the latter can be found in the work by Ignatenko, Surkov & Koltcov (2024), where the potential use of nonclassical entropies for the computation of information gain on RFs under classification and regression tasks was studied. In this work, the nonstandard entropies enabled substantial performance gains (in terms of accuracy) of models, for the application of Rényi, Tsallis, and Sharma–Mittal entropies. The Sharma–Mittal entropy (Akturk, Bagci & Sever, 2007) is defined as

(9) $$S_{SM}(S)=\frac{1}{1-r}[{\left(\sum _{i=1}^n{P}_i^q\right)}^{\frac{1-r}{1-q}}-1],$$and is further capable of retrieving the Rényi entropy for $r\to 1$ and the Tsallis entropy for $q\to r$.

Although these previous works are comparable in motivation to our work, in the sense that the goal is improving the splitting process of tree-based algorithms through the use of nonstandard entropies, the implementation substantially differs. Here, the approach is to develop a new parametric expression that encompasses the classical methods (e.g., retrieving Shannon entropy or Gini index); however, by the nature of additional degrees of freedom, it enables the deduction of additional criteria that may be more suitable for separating data at each split.

Generalized entropy

This section will now introduce the 5-parameter expression for the generalization of entropy, first introduced in the authors’ past work in the scope of data complexity estimation (Costa, Rocha & Ferreira, 2024). Consider a set of probabilities $P=(P_1,\dots ,P_n)\in [0,1{]}^n$ with ${\sum }_{i=1}^n{P}_i=1$, the generalized entropy $\hat{E}$ is defined by

(10) $${\hat{E}}_{{\zeta }_1,{\zeta }_2}^{\alpha ,\beta ,\gamma }(P)=({\hat{\phi }}_{{\zeta }_1,{\zeta }_2}^{\alpha ,\beta ,\gamma }(P)-k_0)(k_1-k_0{)}^{-1},$$where $(\alpha ,\beta ,\gamma ,{\zeta }_1,{\zeta }_2)\in ({\mathbb{R}}_0^{+}{)}^5$ are adequate parameters,

(11) $${\hat{\phi }}_{{\zeta }_1,{\zeta }_2}^{\alpha ,\beta ,\gamma }(P)={\zeta }_2-\ln \left(\sum _{i=1}^n{P}_i^{\alpha }{[-{\zeta }_1-\ln \left(P_i^{\beta }\right)]}^{\gamma }\right),$$and the $q$-logarithm, for any $q\in {\mathbb{R}}_0^{+}\mathrm{\backslash }\{1\}$ and $x>0$, is given by

(12) $$q-\ln (x)=\frac{{\mathit{x}}^{1-\mathit{q}}-1}{1-\mathit{q}}.$$

For $q=1$, the $q-\ln (x)$ coincides with the natural logarithm $\ln (x)$, by computing the limit as $q\to +1$, using the L’Hôpital’s rule, and the fact that $d{x}^{1-q}/dq=-\ln (x)x^{1-q}$. Constants $k_0$ and $k_1$ represent the minimum and maximum theoretical values of $\hat{\mathit{\phi }}$, respectively. Accordingly, $k_0=min(\hat{\phi })$, obtained whenever ${\mathrm{\exists }}_j:P_j=1$. Alternatively, $k_1=max(\hat{\phi })$, occurring whenever each event is equally probable, i.e., $P_i=\frac{1}{n}.$ Table 1 showcases the required parameter re-configuration to achieve a normalized version of the most common generalizations of impurity and entropy used as criteria for splitting in DTs. Proofs for the retrieval of these entropic measures using $\hat{E}$ are given in the Supplemental File ‘Proofs of Known Entropies’. Note, however, that the list shown in Table 1 is not comprehensive, in the sense that there are further entropic definitions that can still be retrieved. Besides generalizing to other entropic estimators, $\hat{E}$ is capable of further obtaining other entropy estimations through parametric variation. In Fig. 1, some of these behaviors are plotted.

To employ the measure $\hat{E}$ as a splitting criterion for DTs in classification tasks, it is first necessary to formulate a target function. Considering (1), the target function will be formulated in a similar way as seen in the work by Ignatenko, Surkov & Koltcov (2024). In the case $S_j$ is the set of data points falling into node $j$, then $I{G}_j=IG(S_j,S_j^L,S_j^R)$, where $S_j^L$ and $S_j^R$ are the subsets which fall into the left and right subtrees, respectively. Therefore, information gain is given by

(13) $$I{G}_j=\hat{E}(S_j)-\sum _{i\in \{L,R\}}\frac{|S_j^i|}{|S_j|}\hat{E}\left(S_j^i\right).$$

As such, when employing this definition of information gain, each of the five parameters accepted by $\hat{E}$ will become a new hyperparameter of the DT. To help guide hyperparameter tuning, a sensitivity analysis was conducted, focusing on parameters $\alpha$, $\beta$, and $\gamma$. Parameters ${\zeta }_1$ and ${\zeta }_2$ were defined as ${\zeta }_1,{\zeta }_2\in \{0,1\}$, as this would simplify the analysis whilst maintaining the generic properties of $\hat{E}$ intact. The analysis was two-stage. First, the Morris screening method (Morris, 1991; Campolongo, Cariboni & Saltelli, 2007) was applied to estimate the mean absolute effect, $\mu \ast$, of each parameter in overall importance, and the standard deviation, $\sigma$, to quantify nonlinearity and interaction effects. Secondly, a Sobol variance-based sensitivity analysis (Sobol, 2001; Saltelli, 2002; Saltelli et al., 2010) was performed to compute both first-order (S1) indices to measure main effects and total-effect (ST) indices to capture the combined impact of main and interaction effects. Both of these methods were applied using the implementation made available in the SALib library (Herman & Usher, 2017; Iwanaga, Usher & Herman, 2022) (SALib version 1.4.7 (https://github.com/salib/salib)).

The comparison of the results yielded from both methods (shown in Figs. 2 and 3) enables the identification of the parameters with strong and stable influences vs. those whose effects are driven primarily by interactions. In this case, no single parameter exerts a predominant influence through main effects, as indicated by the low first-order indexes. While both $\alpha$ and $\gamma$ exhibit the highest first-order influences, these values remain considerably less than the total-effect indexes. Suggesting that the behavior of $\hat{E}$ is mostly driven by interaction between parameters. The results of the Morris analysis for $\beta$ appear to suggest a high degree of influence; however, this assumption is contradicted by the Sobol method, thus suggesting that its interactions are less pervasive or more localized.

Description of computational experiments

The proposed splitting criterion was evaluated for classification tasks across two distinct groups of datasets, namely datasets retrieved from open repositories and synthetically generated datasets. In the case of the former, these were retrieved from both the UCI (https://archive.ics.uci.edu) and OpenML (https://www.openml.org) repositories, and their characteristics are shown in Table 2. In the case of the latter, they were generated using the make_classification function found in scikit-learn (Pedregosa et al., 2011) (scikit-learn version 1.6.1 (https://scikit-learn.org/1.6/api/index.html)), and their description is shown in Table 3. Dataset selection took into account the following requirements: (i) be reasonably broad in terms of application/problem areas; (ii) have a reasonably comprehensive combination between number of features/number of instances across datasets; (iii) validate for both binary and multiclass datasets; (iv) validate for both balanced and imbalanced datasets (in the case of imbalanced datasets, with varying degrees of imbalance); and, (v) contain both numerical (continuous), ordinal, and categorical features across datasets.

Two algorithms were implemented: (i) a (classification) DT, based on the conventional ID3 architecture, which supported both numeric and categorical features and used as its target function the equation defined in Eq. (13); and (ii) a RF based on an ensemble of DTs with feature bagging and voting based on the most common label values. The source code for the algorithms used in these experiments was made available (https://doi.org/10.5281/zenodo.15241909), and was developed with standardization in mind, following the conventions of the popular Python package of scikit-learn, making it compatible with its methods. The fixed parameters for DTs were: 2 samples as the minimum number required to split an internal node; and, minimum impurity decrease set as 0. The foundational DTs for the RFs were constructed with these same fixed parameters.

In terms of non-fixed parameters, for the tests focused on the DTs, their maximum depth was varied as $max\mathrm{\_}depth\in \{10,50,100,200\}$. In the case of RFs, the number of estimators in each forest was varied as $n\mathrm{\_}estimators\in \{10,25,50\}$, and the maximum depth of individual trees was varied as $max\mathrm{\_}depth\in \{10,25\}$. The search space for the parameters in $\hat{E}$ was constructed considering preliminary empirical findings, and was defined as: $\alpha \in \{0,1,2,3\}$; $\beta \in [0,...,3]$, with a step size of $0.1$; $\gamma \in [0,...,3]$, with a step size of $0.1$; ${\zeta }_1\in \{0,1,2,3\}$; and, ${\zeta }_2\in \{0,1\}$. These search spaces were selected based on the prior sensitivity analysis, where the effect of parameter $\alpha$ was comparatively stable while $\beta$ and $\gamma$ produce large variability in effects predominantly through interactions. Consequently, a more coarse grid was used for $\alpha$ to concentrate computational effort where the proposed entropic function $\hat{E}$ is most interaction-sensitive, reducing the risk of missing narrow high-performance regions while keeping total evaluations manageable.

The search for the best-performing hyperparameters for the proposed splitting criterion (i.e., for parameters $\alpha ,\beta ,\gamma ,{\zeta }_1,{\zeta }_2$) was conducted for all combinations of maximum depth and number of estimators in the DTs and RFs algorithms, where applicable. Due to the elevated number of test combinations and datasets, a Bayesian search was conducted over the search space, using the BayesSearchCV class in scikit-optimize (https://scikit-optimize.github.io) (scikit-optimize version 0.10.2), with a set number of 100 iterations (i.e., 100 parameters sampled per dataset per maximum depth, and, in the case of RFs, per number of estimators). Furthermore, these parameters were evaluated through 5-fold cross-validation with the target metric set to weighted F1-score, given by

(14) $$\mathit{F}{1}_{\mathrm{w}\mathrm{e}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}\mathrm{e}\mathrm{d}}=\sum _{i=1}^N\left(\frac{n_i}{{\sum }_{j=1}^N{n}_j}\times F{1}_i\right),$$where N is the number of classes, $n_i$ represents the support for the $i$-th class, and F1_i is the F1-score for the $i$-th class, computed through

(15) $$\mathit{F}1=\frac{2\mathrm{T}\mathrm{P}}{2\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}},$$where $\mathrm{T}\mathrm{P}$ is the number of true positives, $\mathrm{F}\mathrm{P}$ is the number of false positives, and $\mathrm{F}\mathrm{N}$ is the number of false negatives. In addition to the weighted F1-score used for model selection during the hyperparameter search, complementary evaluation metrics were computed, namely Balanced Accuracy, Precision, and Recall. Balanced Accuracy is defined as

(16) $$\mathrm{B}\mathrm{a}\mathrm{l}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{d}\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{1}{N}\sum _{i=1}^N\frac{\mathrm{T}{\mathrm{P}}_i}{\mathrm{T}{\mathrm{P}}_i+\mathrm{F}{\mathrm{N}}_i},$$where $\mathrm{T}{\mathrm{P}}_i$ and $\mathrm{F}{\mathrm{N}}_i$ are the true positives and false negatives for the $i$-th class, respectively. The Precision and Recall for each class are computed as:

(17) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}},$$

(18) $$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}.$$

These metrics were also averaged in a weighted manner according to class support, ensuring that the performance evaluation reflects the influence of class distribution in the dataset.

The usage of the weighted F1-score is uncommon in comparable works found in the literature, which tend to favor reporting accuracy measurements (see, for instance, the experimental methodology in Nowozin (2012) or Ignatenko, Surkov & Koltcov (2024)). However, this is disingenuous as the datasets selected for testing frequently show some degree of class imbalance. For example, the ‘Statlog (Shuttle)’ dataset, used in testing in multiple comparable works (including the two previously mentioned), is a seven-class set with the majority class representing nearly 80% of instances. Hence, the weighted F1-score is selected as the performance evaluation metric for this work, and thus any mention of F1-score will be regarding its weighted implementation. The weighted version was selected (instead of macro or micro extensions) as a compromise, somewhat attenuating the effect of very low support classes negatively skewing results, while still providing enough sensitivity for penalizing misclassifications in imbalanced datasets.

The introduced parametric entropies were benchmarked against the known splitting criteria given by the Gini index, Shannon entropy, Rényi entropy, and Tsallis entropy, as these are the most utilized in comparable works. The benchmarks were obtained using the same tree structures and parameters as the introduced method. In the case of Rényi, parameter $\alpha$ in Eq. (6) was set as $\alpha =2$, and, in the case of Tsallis, parameters $k$ and $q$ in Eq. (8) were set as $k=1$, and $q=2$.

Results of computational experiments

To aid in interpreting results, in this section only a summary presentation is given (shown in Table 4), where, for each result obtained, two additional metrics were computed: Lowest Improvement (LI), and Highest Improvement (HI). Respectively, these correspond to the lowest and highest percentual gain in F1-scoring of the parametric entropies obtained using $\hat{E}$ compared to the known entropic measures. The complete results for the computational experiments can be seen in the Supplemental File ‘Complete Results for Proposed Method’, these include: results for DTs (shown in Tables S1, S2, S3 and S4); results for RFs using a maximum depth of 10 for the DTs (shown in Tables S5, S6, and S7); and, results for RFs using a maximum depth of 25 for the DTs (shown in Tables S8, S9, and S10).

To more effectively assess the method’s performance, the parameter combination yielding the highest weighted F1-score for each dataset in the hyperparameter search was selected. This optimal configuration was subsequently compared with the known entropy values. The evaluation was carried out using a class-wise stratified 5-fold cross-validation procedure. In this approach, the dataset is split into five equally sized folds while preserving the proportion of classes in each fold. For each iteration, the model is trained on four folds and tested on the remaining one. This process is repeated five times so that each fold serves once as the test set, and the results are then averaged to obtain the final performance metrics. The results are presented in terms of the Highest Improvement and Lowest Improvement observed. For the DT algorithms, these results are shown in Tables 5, 6, and 7, which correspond to Balanced Accuracy, Precision, and Recall, respectively. The equivalent results for the RF algorithms are reported in Tables 8, 9, and 10.

Discussion

Through the analysis of Table 4, it is possible to see that the introduced method shows improvements in comparison with classical entropies across almost all tree structures and datasets. Even with a relatively reduced number of search iterations (100 iterations), we were able to frequently outperform existing methods. These improvements are less noticeable, as expected, in the case of the weaker learner (DTs), where a higher proportion of null LI is found. Nonetheless, it must be considered that metric LI provides a worst-case scenario outcome, where comparisons are always made with the best performance of the existing entropic measures. Generally, the combination of the proposed method with RFs yields a more substantial increase in performance, with a reduced number of null LI, with the notable exception being the ‘wilt’ dataset, in which almost all methods showed similarly high performance. Moreover, in terms of HI, it is possible to see improvements of over 40%, especially in the case of the stronger RF models, particularly in the case of the ‘Wine Quality’.

In general terms, the highest performance gains seem to occur on multiclass and imbalanced datasets (e.g., ‘Wine Quality’, ‘Statlog’), where the choice of performance metric (weighted F1-score) would be most unfavorable. Another example can be seen in the case of the synthetic datasets, namely ‘synth_2’ and ‘synth_5’, which are the two synthetic sets with the highest performance increase, and, simultaneously, the only two multiclass synthetic datasets, albeit with balanced classes. Regarding data types, the performance of the proposed method does not seem to suffer negatively. For instance, in datasets with mixed datatypes (e.g., ‘Students’), the introduced criterion is still able to outperform classical methods. The results for the remaining metrics, namely Balanced Accuracy, Precision, and Recall (seen in Tables 5-10) generally show a lesser improvement of the proposed method, compared to the standard methods. This is, however, expected as these metrics do not accurately reflect the large data imbalance that exists in multiple of the tested datasets. Nevertheless, the results provide some additional insights into the method’s behavior. Specifically, the gains in weighted F1-score appear to stem not only from a modest overall improvement in classification balance, as reflected in the slight increases in Balanced Accuracy, but also from an improved detection of minority classes, as indicated by the relative improvements in Recall for underrepresented categories.

The largest advantage of the proposed method as a splitting criterion is that it never introduces a performance penalty, and, at worst, it is only capable of equalling the classical measures. This is more clearly shown in Tables 11 and 12. When using the ROPE value of 1% (as suggested in Benavoli et al. (2017) and later applied in Hernández et al. (2021) for the comparison of split criteria), our method is seen to win against all other methods in the case of DTs and RFs. When increasing ROPE to 2%, we are still able to significantly win half the tests when using the RFs. Note that draws were never significant and that the proposed method never lost against classical ones. Furthermore, the statistical results in Table 12 show that using $\hat{E}$ as a splitting criterion is particularly advantageous on shallow tree ensembles. A possible interpretation for this occurrence is that the low maximum depth, in combination with our criteria $\hat{E}$, makes it so that the classifier can more easily generalize to unseen data.

Although the proposed method suffers from implementation challenges, since the foundational structure for the devised DTs/RFs is not on the level of optimization as seen in established works such as scikit-learn, resulting in considerably higher computing times, it could nonetheless be integrated through the addition of the proposed split criterion component within the algorithms. As such, there had to be a compromise in terms of the number of sampled hyperparameters (both from the introduced parametrization and for conventional DT/RF parameters). Moreover, compared with, for instance, Gini, the computational cost of $\hat{E}$ is much greater due to the introduction of logarithmic functions. This, however, seems like a reasonable middle ground considering the added flexibility introduced by our method. This can be of particular value when in combination with solutions such as that discussed in Loyola-González, Ramírez-Sáyago & Medina-Pérez (2023), where the dynamic selection of splitting criteria at each split could greatly benefit from a unified parametric expression that could be easily searched through, for instance, a genetic search algorithm.

Conclusions

Tree-based classifiers are widely used in practical machine-learning applications. The DT, in particular, despite its simplicity, can have reasonable performance while being easily interpretable. It is also the foundational component for other stronger learning models, such as RFs. These factors still motivate research into the further development of DTs. The work described in this article focuses on the data-splitting process, which is one of the key algorithmic mechanisms of DTs. For this, we introduced a 5-parameter expression that aims to generalize entropies traditionally employed as target functions to partition data at DT nodes. This expression, $\hat{E}$, is capable of, for instance, retrieving the Gini impurity, Shannon entropy, Rényi entropy, and Tsallis entropy. However, because of its flexibility, other entropic estimators can also be extracted from parametric manipulation.

To test its applicability as a splitting criterion, a target function was formulated based on $\hat{E}$ and employed in both DTs and RFs. This criterion was then validated against the most common information-theoretic measures of entropy on multiple datasets (both synthetic and derived from open data repositories), using 5-fold cross-validation and parameter tuning through a reduced 100-iteration Bayesian search. From the preliminary results, two main conclusions can be drawn: (i) performance improvements could reach upwards of 40% in F1-score compared to classical measures of entropy; (ii) in the worst-case scenario, no performance penalization would be introduced through the use of $\hat{E}$ (it would only be equivalent to the best method). To validate the latter of these claims, an additional Bayesian statistical analysis was conducted (signed-rank test), showing that for the conventional ROPE value of $1\%$, our method only produced significant wins against comparable alternatives. Furthermore, our method never suffered from any significant loss, assessing its assurance to worst-case match performance. These findings underscore the importance of careful selection of a splitting strategy.

As future research paths, multiple potential advancements can be identified, but perhaps the most relevant are: a deeper study into the influence of each parameter in $\hat{E}$ in the overall performance of the classifier, guiding prospective practitioners and reducing computational costs of extensive search spaces; the adaptation of this target function into regression problems; the combination of this method with the dynamic selection of split criteria (as discussed in Loyola-González, Ramírez-Sáyago & Medina-Pérez (2023)), taking further advantage of the flexibility introduced by $\hat{E}$; and, the application of the proposed method in streaming data scenarios through the incorporation of $\hat{E}$ into the internal operating structure of Hoeffding Trees.

Supplemental Information