On the interpretability of fuzzy knowledge base systems

Introduction

In recent years, computational intelligence methods have experienced a great interest in interpretable and eXplainable Artificial Intelligence (XAI) (Mencar & Alonso, 2018; Fernandez et al., 2019). Techniques for XAI can be model-independent (i.e., they can be applied to any AI algorithm) or model-specific (i.e., they can only be applied to a specific AI algorithm). Furthermore, they can be ante-hoc (transparent or “white box/glass box” approaches) or post-hoc (divided into global explanations or local explanations) explanatory methods (Knapič et al., 2021). In particular, ante-hoc methods can be explained by design or inherently explained (referred to as transparent approaches). Transparent approaches are model-specific and include linear and logistic regression, decision trees, k-nearest neighbors, fuzzy inference systems, rule-based learners, general additive models, and Bayesian models. In recent years, fuzzy rule-based systems (FRS) have gained significant attention in explainable AI (XAI) as an ante-hoc approach. The main components of an FRS are a knowledge base (KB) and an inference module. KB contains all fuzzy rules within a rule base (RB) and the definition of fuzzy sets in the database. The inference module includes a fuzzification interface, an inference system, and a defuzzification interface (Di Nardo & Ciaramella, 2021; Camastra et al., 2015). However, it is crucial to emphasize that FRSs need to remain simple and comprehensible, as they are not inherently interpretable (Mendel & Bonissone, 2019; Fernandez et al., 2019). To ensure that fuzzy rule-based systems represent knowledge that is easily understood by humans, several key aspects must be taken into account. These include, among other things, the compactness of the rule base and the semantic clarity of the fuzzy partition (Jara, González & Pérez, 2020; Mendel & Bonissone, 2019). In addition, FRSs must be properly designed to achieve the desired trade-off between accuracy and explainability for the problem at hand. This work presents a fuzzy rule base minimization algorithm based on rough set theory and a greedy strategy. The problem of finding useless attributes for correct classification is NP-hard and it is intractable when the number of attributes is not small. Therefore, algorithms that provide suboptimal approximate solutions need to be explored.

The proposed fuzzy rule base minimization algorithm called Reducing Rules and Conditions (RRC), provides a greedy approximate solution to the problem of identifying and removing attributes that are irrelevant to correct classification from a set of fuzzy rules. RRC roughly consists of five main stages, as described in Fig. 1. To evaluate a rule pattern (a rule pattern of a rule $\mathcal{R}$ has an antecedent composed of a subset of the conditions of $\mathcal{R}$ and the same consequent of $\mathcal{R}$) in the search phase, a property, i.e., efficiency, is explicitly defined and associated with each rule pattern. The definition of efficiency reflects the main goal of the RRC algorithm, i.e., to search in each subsequent set for the rule patterns with the smallest length of the antecedent, which covers the largest number of rules while having the least overlap with other subsequent sets. The outline of the article is as follows. The section Basics of Rough Set Theory recalls the fundamentals of Rough Set Theory used in the RRC algorithm; in section Reduce Rules and Conditions Algorithm, the RRC algorithm is described; some experimental results are reported in the Experimental Results section, and finally, some conclusions are drawn in the Conclusions section.

Basics of rough set theory

Since the proposed algorithm utilizes approaches and concepts derived from Rough Set Theory (Pawlak, 1982), the foundations of Rough Set Theory are briefly reviewed to improve the readability of the article.

Reduce rules and conditions algorithm

As shown in the section Reduction of attributes, the problem of finding the useless attributes for the correct classification of objects is intractable when the number of attributes is not small. Therefore, in most real-world applications it is necessary to look for algorithms that provide suboptimal, approximate solutions to the aforementioned problem. The Reduce Rules and Conditions (RRC) algorithm provides a greedy approximate solution to the problem of identifying and removing the attributes, from the set of fuzzy rules, that are irrelevant to the correct classification. Before describing the proposed RRC algorithm, the following definition needs to be introduced.

More simply, as shown in Fig. 2, a rule pattern of a rule $\mathcal{G}$, is a rule that has an antecedent composed of a subset of the conditions of $\mathcal{G}$ and the same consequent of $\mathcal{G}$.

To evaluate a rule pattern within the algorithm search, the property of the rule pattern, the efficiency, is suitably defined in this work. Specifically, given a rule pattern $\mathcal{P}$ and a consequent $\mathcal{D}$, the efficiency of the rule pattern $\eta (\mathcal{P},\mathcal{D})$, is given by:

(2) $$\eta (\mathcal{P},\mathcal{D})=\frac{|\underset{\mathrm{\Sigma }}{sup}(\mathcal{P})\cap \mathcal{R}(\mathcal{P},\mathcal{D})|}{{(|\underset{\mathrm{\Sigma }}{sup}(\mathcal{P})|-|\underset{\mathrm{\Sigma }}{sup}(\mathcal{P})\cap \mathcal{R}(\mathcal{P},\mathcal{D})|)}^{\mathcal{L}}+\lambda {(\mathcal{P})}^K}$$where $su{p}_{\mathrm{\Sigma }}(\mathcal{P})$ is the superior of set of rules $\mathrm{\Sigma }$ that match the antecedent of the rule pattern $\mathcal{P}$, $\mathcal{R}(\mathcal{P},\mathcal{D})$ is the set of rules with consequent $\mathcal{D}$ that match the antecedent of the rule pattern $\mathcal{P}$, and $\lambda (\mathcal{P})$ is the length of the antecedent of the rule pattern $\mathcal{P}$. The optimal values for $\mathcal{L}$ and K can be determined by model selection techniques (Hastie, Tibshirani & Friedman, 2009). However, it has been found empirically that typical good values for $\mathcal{L}$ and K are 10 and 2, respectively.

The definition of efficiency reflects the RRC algorithm rationale, i.e., for each consequent, searching for rule patterns with the smallest antecedent length, covering the largest number of rules, and having, at the same time, the minimum overlap with other consequents. The greater the number of rules covered by the rule pattern for a given consequent, the better the rule pattern is because it can represent a larger number of rules. Therefore, this value must be proportional to the efficiency, thus justifying the numerator of the Eq. (2).

The denominator expresses the number of rules covered by the rule pattern for other consequents, and the length of the rule pattern, raised to $\mathcal{L}$ and K, respectively. These two values must be as small as possible since the overlap with other consequents must be minimal; in addition, the antecedent length of the rule pattern should be as small as possible. In Fig. 3 an example of the efficiency of a rule pattern computed in a knowledge base is presented.

In practice, RRC takes as input three parameters, R, C, T, namely, the minimum efficiency, the minimum rule coverage, and the number of iterations, and a fuzzy knowledge base $\mathcal{F}$. The algorithm is composed of five steps, which in turn is going to deepen in the next paragraphs:

1. Building decision tables from fuzzy rules;

2. Sorting attributes by their significance;

3. Building prefix tree;

4. Rule pattern searching;

5. Sorting rule patterns and building the minimum set of rules.

Sorting attributes by their significance

From the decision table, the relevance of each attribute (i.e., for each fuzzy relation) is computed. For this purpose, the relevance of the attribute is measured by its significance, as defined in the Significance of attributes section. Then, the attributes are sorted by their significance (see Fig. 4). The sorting of the attributes allows the implementation of a greedy strategy in the next steps of the algorithm.

Building prefix tree

In this phase, a prefix tree is built. The prefix tree is built by processing one rule at a time. For each attribute, a node is created, which becomes a child of the node of the previous attribute and a parent of the node of the next attribute. The first attribute becomes the child node of the root tree, i.e., an empty node, since it has no attribute. The tree has the following properties, namely, that all descendants of a node share the prefix associated with the node, and this implies that two rules having the same initial condition, share the same node associated with that condition. An example of a prefix tree is shown in Fig. 5.

Rule pattern searching

Before describing the rule pattern search, it is worth introducing a new definition and presenting a theorem that provides the theoretical motivation for the search algorithm.

Definition 2 Let E be an edge that connects two nodes $x$ and $y$. Let $\mathcal{S}$ be a set of rules passing through E. We say that E is optimal if

(3) $$\{(x,y)\in \mathcal{S}\times \mathcal{S}:consequent(x)\ne consequent(y)\}=\mathrm{\varnothing }.$$

In other words, an edge is optimal only if it is connected to a single consequent. In Fig. 5, for instance, there are four optimal edges: A-B, B-C, A-C, C-B, while the remaining two edges are not optimal.

Theorem 1 Let $P=(E_1,\dots ,E_N)$ be a path, where $\{E_i{\}}_{i=1}$ are edges. If the last edge $E_N$ is an optimal edge then P is an optimal path.

The proof of the theorem is given in Appendix A (Proof of Theorem 1). Now, it comes to the description of the search algorithm. The search cannot be exhaustive since the number of possible rule patterns in a fuzzy knowledge base can be very high, even huge. Therefore, it adopts a greedy strategy and it focuses only on extracting a subset of the rule patterns with the highest efficiency. The search algorithm uses the prefix tree as a decision tree, constructed in the previous stage of the RRC algorithm, and performs a left-to-right depth-first search (DFS) visit of the tree for each decision (Cormen, Leiserson & Rivest, 2009). The search algorithm is greedy because the attribute importance of nodes decreases from left to right. For each explored node in DFS, there are four possible cases:

• The current path, i.e., the rule pattern, is optimal. Therefore, it is added to the list of candidate rule patterns and the search is stopped since adding another edge, by Theorem 1, can only decrease the efficiency of the path.

• The rule pattern is not optimal, and its efficiency is $\eta >R$ (i.e., greater than the minimum efficiency), and the number of rules covered by the consequent is $\nu >C$ (i.e., greater than the minimum coverage). The path is added to the list of candidate rule patterns, and the search continues.

• The rule pattern is not optimal, and its efficiency is $\eta \le R$, while $\nu \ge C$. The search continues but the path is not added to the list of candidate rule patterns.

• The rule pattern is not optimal, its efficiency is $\eta \le R$, and it results $\nu \le C$. In this case, the DFS visit stops and the rest of the tree is visited.

It should be emphasized that the search algorithm is not applied directly to the complete tree derived from the initial knowledge base, but rather, iteratively, to a sequence of its reductions, since the application of the search algorithm to the complete tree can become computationally intractable when the total number of attributes of the knowledge base is large. Therefore, for each iteration, a reduction of the complete tree is computed and the search algorithm is executed. The number of iterations is given by the input parameter T. The reduction of the full tree is simply obtained by randomly discarding some attributes from the decision table, and constructing a pruned prefix tree, as shown in Fig. 6.

Computational complexity of rrc algorithm

This section discusses the computational burden of RRC, focusing only on the worst case. Since the RRC algorithm consists of five different stages, it is necessary to evaluate the complexity of each single stage to get the overall computational complexity.

The first stage builds the decision table. The computational complexity of this operation is $O(MN)$, where M and N are the maximum number of attributes of a rule and the number of rules in the knowledge base, respectively. In the second stage, i.e., sorting attributes by their importance, the computation of the importance of the attributes (i.e., fuzzy relations) and their sorting are performed. The complexity of the first operation is still $O(MN)$. Since the sorting of the attributes requires $O(M\log M)$, the complexity of the stage is $O(MN+M\log M)$.

The third stage requires the construction of the prefix tree, whose computational complexity is again $O(MN)$.

The fourth stage, i.e., rule pattern search, requires that the rule pattern search be repeated T times. Each iteration involves a DFS visit of a reduced prefix tree. The iteration requires a complexity of $O(MN)$, for the construction of the reduced prefix tree, and a complexity of $O(MND)$, where D is the total number of decisions of the knowledge base. Therefore, the complexity of each iteration is $O(MN+MND)$. Since each iteration is repeated T times, the complexity of the fourth stage of the RRC algorithm is $O(T(MN+MND))$, or more simply $O(TMN(D+1))$.

In the fifth stage, the candidate rule patterns are sorted and the first rule patterns covering the initial knowledge base are extracted.

The sorting requires $O(TMN(D+1)\log TMN(D+1))$, while the complexity of the extraction is $O(N)$. Then, the complexity of the stage is $O(TMN(D+1)\log TMN(D+1))$+ $O(N)$. Since the latter term is negligible w.r.t the former, the complexity of the stage is $O(TMN(D+1)\log TMN(D+1))$.

Therefore, the overall complexity of RRC is the sum of the complexities of its component stages, namely:

$$\begin{array}{rl}& O(MN)+O(MN)+O(M\log M)+O(MN)+O(TMN(D+1))\\ & +O(TMN(D+1)\log TMN(D+1)),\end{array}$$and, since the first five terms are negligible w.r.t the last one, it can be written as $O(TMN(D+1)\log TMN(D+1)$. Finally, it is worth noting that RRC has lower complexity than Ripper Algorithm, in terms of the number of rules N. Ripper Algorithm has complexity $O(N{\log }^{2}N)$, while RRC has complexity $O(N\log N$).

Experimental results

The proposed algorithm has been validated on four UCI benchmarks, namely, mushroom (Schlimmer, 1987), breast-cancer (Michalski et al., 1986), nursery (Olave, Rajkovic & Bohanec, 1989), primary-tumor (Cestnik, Kononenko & Bratko, 1986)¹ , and was compared with Ripper (Cohen, 1995), Part (Frank & Witten, 1998), Lem2 (Grzymala-Busse, 1992) and Evolutionary based (Gen & Lin, 2023) algorithms (Camastra et al., 2024). See https://github.com/angelociaramella/IFKBS for raw data and code (JavaScript). The benchmark characteristics are summarized in Table 3. The performance of the algorithms was measured in terms of the number of rules extracted, coverage, and accuracy. Coverage measures the percentage of the original knowledge base rules, covered by the set of rules generated by the compression algorithm. Accuracy measures the percentage of the correct classification of the set of rules generated by the compression algorithm.

The results on the mushroom, breast cancer, nursery, and primary-tumor benchmarks are reported in Tables 4–7, respectively. Experimental comparisons, on the benchmarks, with other recent methods (Yang et al., 2021; Chen et al., 2023) are not reported since their results are not reproducible due to the unavailability of their implementation. Moreover, these methods cannot theoretically compared with RRC algorithm, since their computational complexities are not declared.

Finally, the RRC algorithm was evaluated on the TERA knowledge base (Camastra et al., 2014, 2015, 2024; https://github.com/angelociaramella/IFKBS.git). TERA is a fuzzy decision system for environmental risk assessment related to the cultivation of genetically modified plants. The TERA knowledge base consists of 6,215 rules and 26,959 attributes. Once applied to the TERA knowledge base, RRC extracted 2,562 rules and 9,319 conditions, whose compression rates are $59.8\mathrm{\%}$ and $65.4\mathrm{\%}$, respectively, guaranteeing a $100\mathrm{\%}$ of both coverage and accuracy.

Conclusions

This article presented the Reducing Rules and Conditions (RRC) algorithm, a fuzzy rule base minimization algorithm based on rough set theory and a greedy strategy. Fuzzy set theory was used to evaluate the importance of attributes, while the greedy strategy was used to construct a prefix tree, which is employed as a decision tree traversed by a left-to-right depth-first search (DFS) visit. Several knowledge bases from UCI benchmarks were used to evaluate the behaviour of the proposed approach and to compare its performance with some competing methods from the literature. In any case, the results are promising, as they lead to more compact knowledge bases compared to the competitors or, at least, to higher coverage and accuracy percentages. The latter results are a prerequisite for building explainable decision support systems. Nevertheless, future work is needed to establish RRC as one of the reference methods for developing interpretable inference systems. In particular, emphasis will be placed on the theoretical background of the method (e.g., the search of diverse rule pattern efficiency measures), and the investigation of the explainability of the obtained fuzzy rules will also require more in-depth attention by comparing recent results in this context (Gallo et al., 2020).

Appendix