Quick mining in dense data: applying probabilistic support prediction in depth-first order

Motivation

Frequent itemset mining is a highly computational and memory-intensive task. It needs to find all the frequent itemsets and their possible combinations that are also frequent. FIM also needs to store the transaction data in memory associated with these discovered frequent itemsets. This problem becomes even more severe when FIM is deployed to large dense datasets. A dense dataset is one in which every frequent item is assumed to be present in almost every transaction (Fournier-Viger et al., 2017; Luna, Fournier-Viger & Ventura, 2019). One reason for this is that the number of frequent itemsets reaches into billions even for an average-sized dense dataset with only a few dozen frequent items.

To make FIM algorithms more efficient and scalable, different researchers have proposed various methods. One of these methods was to propose efficient data structures to speed up FIM on dense datasets, but the constant challenge introduced by advances in computing technology is the tremendous growth in data. Employing maximal frequent itemsets (MFI) mining in dense datasets is another solution. While the support of an itemset indicates its importance in the FIM domain, with the MFIs, we can only identify all the frequent itemsets–we cannot rank them based on their support because we lack the support information for each individual frequent itemset.

Some researchers have attempted to restrict the use of transactional data and calculate an approximate value for the support of itemsets. Though the downside of these algorithms is that they do not guarantee to generate all possible frequent itemsets, the resulting loss is marginal. These algorithms have employed different techniques to limit the transactional data used to derive the support of an itemset. For example, some researchers have used sampling techniques (Li et al., 2016), hashing methods (Zhang et al., 2018), filtering techniques (Abbasi & Moieni, 2021), and clustering algorithms (Fatemi et al., 2021).

In one such recent effort, a probability-based technique, called probabilistic support prediction model (PSPM), was proposed in Sadeequllah et al. (2024). This technique is used to probabilistically predict the support of an itemset. This article also proposes a FIM algorithm, called Probabilistic Breadth-First (ProbBF), that incorporates the PSPM technique. This algorithm is different from other FIM algorithms in that it does not use any additional parameter other than the minimum support threshold. Additionally, it uses zero transaction data in its support approximation. These characteristics render the ProbBF algorithm more efficient than other FIM algorithms, both approximate and exact. It is also worth noting that ProbBF is the first approximate FIM algorithm designed for dense data, whereas other approximate algorithms typically perform well with sparse data only.

However, problem with the ProbBF algorithm is that it is a breadth-first algorithm, and it is established that breadth-first algorithms are less efficient and less scalable than depth-first algorithms when deployed to dense data. This is due to the fact that the memory requirements of breadth-first and depth-first algorithms are respectively $O\left(w\right)$ (where $w$ is the maximum width of the search space tree) and $O\left(h\right)$ (where $h$ is the maximum depth of the search space tree). For dense data, $w$ is much more larger (order of magnitudes) than $h$. High memory requirements could also slow an algorithm due to the reasons such as low utilization of locality of reference, thrashing, and aggressive garbage collection. Although, a pure depth-first algorithm also suffers from slow runtime due to large number of candidate generation. A better compromise is a hybrid search strategy used by many breadth-first FIM algorithms (e.g., MAFIA (Burdick et al., 2005), Genmax (Gouda & Zaki, 2005), NegFIN (Aryabarzan, Minaei-Bidgoli & Teshnehlab, 2018), SelPMiner (Bai et al., 2019)).

Another important shortcoming of the ProbBF algorithm is its lack of support for additional pruning techniques beyond apriori pruning. Many modern FIM algorithms designed for dense datasets incorporate a highly efficient pruning technique initially used by Maxminer and later known by various names such as Parent Equivalence Pruning (PEP) in MAFIA (Burdick et al., 2005), Genmax (Gouda & Zaki, 2005), and SelPMiner (Bai et al., 2019), Hypercube Decomposition in LCM (Uno, Kiyomi & Arimura, 2004), and Promotion Pruning in PrePost+ (Halim, Ali & Khan, 2020), dFIN (Xun et al., 2021), and negFIN (Aryabarzan, Minaei-Bidgoli & Teshnehlab, 2018). This technique prunes entire subtrees rooted at frequent itemsets whose support equals that of their parent frequent itemsets, significantly enhancing the runtime efficiency of FIM algorithms on dense data.

Furthermore, ProbBF employs a set-enumeration tree to store frequent itemsets, which is memory inefficient. The nodes in this tree store pointers to both their children and parent nodes, with the frequent prefix distributed across multiple nodes. A more memory-efficient technique is needed to improve memory utilization in the proposed algorithm.

The ProbDF algorithm, proposed in this article, is an efficient depth-first algorithm that addresses all the shortcomings of ProbBF (Sadeequllah et al., 2024). ProbDF utilizes a novel depth-first traversal method that meets the requirements of the PSPM technique, a cornerstone of ProbBF. It identifies all frequent children itemsets of an itemset using a breadth-first approach and subsequently traverses these children itemsets in a depth-first manner. This algorithm also introduces a technique to incorporate the Promotion Pruning concept. Lastly, it proposes a single vector technique to store all frequent suffixes.

Contributions

The main contributions of this article are as follows:

• 1)

  This article proposes a depth-first algorithm ProbDF with a novel search strategy that ensures the support of all ( $k\text{-}1$) itemsets of a $k$-itemset be already known before evaluating the support of the $k$-itemset. This is an important requirement of the PSPM model.

• 2)

  This algorithm introduces a time and space efficient single vector technique to store a common $k$-1 size prefex for all the $k$-size itemsets currently being explored. This is contrary to the other algorithms that store $k$-1 frequent items prefix with every $k$-size frequent itemset.

• 3)

  This algorithm also introduces the concept of promotion pruning to approximate FIM algorithms to enhance its memory and runtime efficiency.

The structure of the article proceeds as follows: “Related Work” offers a review of the most pertinent literature to our proposed solution in the field of FIM. “Problem Setting and Preliminaries” introduces preliminary concepts relevant to frequent itemset mining. “Probabilistic Support Prediction Model” outlines the PSPM technique discussed earlier. Following this, “The Proposed Algorithm, ProbDP” unveils our proposed algorithm, ProbDF. “Experimental Results” delves into a comprehensive analysis of ProbDF’s performance against state-of-the-art FIM algorithms using real-world benchmark datasets. Lastly, “Future Work” provides the article’s concluding remarks.

Problem statement and definitions

Consider a set of $n$ transaction items, denoted as $I=\left\{i_1,i_2,...,i_n\right\}$, and a collection of $m$ transactions, denoted as $T=\left\{t_1,t_2,...,t_m\right\}$, where each transaction $t_i$ is a subset of $I$. Let $X\subseteq I$ be an itemset of size $k$, referred to as a $k$-itemset. For example, the set $\left\{A,B,C\right\}$ is a 3-itemset, which can be abbreviated as $ABC$.

If $Y\subseteq T$ and for every $y\in Y$ and $X\subseteq y$, then the support of $X$, denoted $\sigma \left(X\right)$, is defined as $\sigma \left(X\right)=\left|Y\right|/\left|T\right|$, where $\left|Y\right|$ is the support count of $X$. This implies that $\sigma \left(X\right)$ is the probability that the itemset $X$ appears in a randomly selected transaction from $T$. An itemset $X$ is considered frequent if $\sigma \left(X\right)\ge min\mathrm{\_}sup$, where $min\mathrm{\_}sup$ is a user-defined threshold for support.

An association rule $A\Rightarrow B$ represents the probability that a random transaction $t\in T$ contains both itemsets $A$ and $B$, given that $A\in I$, $B\in I$, $A\ne Ø$, $B\ne Ø$, and $A\cap B=Ø$. The support for the rule $A\Rightarrow B$ is the joint probability of $A$ and $B$, denoted as $P\left(A\cap B\right)$, which is computed as $\sigma \left(AB\right)$. The confidence of the rule $A\Rightarrow B$ is the conditional probability $P(B|A)$, defined as $\sigma \left(AB\right)/\sigma \left(A\right)$.

In a standard association rule mining process, all potential rules meeting specified criteria—such as user-defined thresholds for support and confidence—are generated for a given dataset. The most computationally demanding aspect of this process is frequent itemset mining. Determining the frequency of an itemset $X$ involves inspecting all transactions containing $X$, known as the support count of $X$, which entails significant computational expenses. In this article, we use a technique presented in the following section to approximate this value.

Transactions data format

The transaction dataset comprises transactions, each assigned a unique transaction ID (TID) along with associated transaction items. FIM algorithms utilize either horizontal or vertical formats of this data for support count calculations of itemsets. For instance, Table 2 illustrates a transaction dataset of 10 transactions, depicted in horizontal format on the left and vertical format on the right. Calculating the support count of an itemset in the horizontal format involves tallying all transactions that include the itemset. Conversely, in the vertical format, the intersection operation is applied to the TID sets of the items constituting the itemset, with the support count being the size of the resulting TID set. ProbDF is tailored to utilize arrays that exclusively store the counts of these TIDs for all itemsets of size two.

Enforcing limits on predictions

If $X=\left(A\cup B\right)$, $Y=\left(A\cup C\right)$ and $Z=\left(B\cup C\right)$, according to the anti-monotone property of the support finding process, the ProbDF introduces the upper bound ( $Ub$) on support prediction as follows.

(10) $$\sigma \left(A\cup B\cup C\right)\le min\left(X,Y,Z\right)=Ub.$$

Similarly, a lower bound on the support of any 3-itemset could be defined using the following equation.

(11) $$\sigma \left(\left\{a,b,c\right\}\right)\ge \sigma \left(\left\{a,b\right\}\right)+\sigma \left(\left\{a,c\right\}\right)-\sigma \left(\left\{a\right\}\right)$$

This equation was first used in Max-miner (Bayardo, 1998). Consequently, a lower bound ( $Lb$) on the support prediction of a 3-itemset could be defined using the following equation.

(12) $$\sigma \left(A\cup B\cup C\right)\ge max\left(\begin{array}{c}\sigma \left(X\right)+\sigma \left(Y\right)-\sigma \left(A\right),\\ \sigma \left(X\right)+\sigma \left(Z\right)-\sigma \left(B\right),\\ \sigma \left(Y\right)+\sigma \left(Z\right)-\sigma \left(C\right)\end{array}\right)=Lb.$$

The ProbDF clipps the predicted support on any of these theoratical bounds whenever crossed. These bounds also help to establish a limit on the maximum error. Let $\vartheta$ represents this maximum error given as follows.

(13) $$Error\le \vartheta =Ub-Lb-1.$$

In the above equation bounds the error $\left[0,\vartheta \right]$). The value $-1$ in the above equation represents a unique situation, which is when $\sigma \left(A\cup B\cup C\right)=min\left(X,Y,Z\right)$. At this point $Lb=Ub$. It has been shown in Sadeequllah et al. (2024) that these bounds improves the predictions of the ProbDF greatly.

Introducing promotion pruning in ProbDF

The parent equivalence pruning (PEP) feature, first introduced in Bayardo (1998) and elaborated in detail in Sadeequllah et al. (2024), has been tuned to the needs of frequent item mining in Aryabarzan, Minaei-Bidgoli & Teshnehlab, (2018), Deng, (2016) under the name promotion method. According to this method if $F$ is a $k$-size frequent itemset and $S$ is a set of items representing the frequent 1-item extensions of F such that $\sigma \left(F\right)=\sigma \left(F\cup i\right)\mathrm{\forall }i\in S$, then $\sigma \left(F\right)=\sigma \left(F\cup j\right)\mathrm{\forall }j\in \mathcal{P}\left(\mathcal{S}\right)$, where $\mathcal{P}\left(\mathcal{S}\right)$ is the power set of $S$. If $M=\left\{F\cup j|j\in \mathcal{P}\left(\mathcal{S}\right)\wedge \mathcal{s}\mathcal{i}\mathcal{z}\mathcal{e}\left(\mathcal{j}\right)>1\right\}$, it is important to note that $M$ is the set of frequent itemsets that are directly generated (i.e., bypassing the candidate generation and test strategy of the FIM algorithms). The experiments show the promotion method significantly improves the time and space complexity of the FIM algorithms (Aryabarzan, Minaei-Bidgoli & Teshnehlab, 2018; Deng, 2016). It prunes complete branches of the search space rooted at promoted items (the frequent items in set $S$ are known as promoted items). The Algorithms 4 and 5 are modified versions of the Algorithms 3 and 4, respectively that incorporate the promotion method in ProbDF algorithm. The changes in these algorithms from their previous versions are highlighted in bold-face font. The promoted items in these algorithms are stored in a new vector $p\mathrm{\_}items$ in every node.

Example: The ProbDF algorithm has been applied to the dataset shown in Table 2. This dataset is passed as an argument to the main algorithm, probDFAlgo (Algorithm 2), along with the support threshold of two transactions. For this support threshold, all items present in the dataset are already frequent. The probDFAlgo first calls the getFrequentItems routine and receives in return the list of frequent items, arranged in ascending support order, as $F1$. For the data in Table 2, the $F1$ is $\left\{S,T,W,X,Y,Z\right\}$.

The probDFAlgo starts traversing the search space from line 5 to line 9, creating the child nodes of the implicit root node for every frequent item in $F1$ as a frequent child. This is shown in Fig. 3A. The support of the implicit root node is set at |Ɗ| because we use the diffset support computations method (Zaki & Gouda, 2003). For the data in Table 2, |Ɗ| is 10. Every child node also has a vector to store support values of all its possible children, and the probDFAlgo determines these values from the transactions database Ɗ from lines 10–19. Lines 22 and 23 perform further exploration of the nodes in Fig. 3A. The probDFAlgo calls the genFItems routine for this purpose.

The genFItems recursively explores the nodes it receives as arguments in depth-first order. It creates a child node for frequent children only. The support of the children of the top-level nodes (i.e., nodes listed in Fig. 3A) was calculated from the transaction database by the calling routine probDFAlgo. Therefore, the top-level nodes and their children nodes are shown as round nodes to indicate their support is calculated from the transaction database. These nodes are shown in Fig. 3B. The later nodes down the tree, starting from level-3, are shown as rectangular nodes to indicate their support is approximated using PSPM. It is worth noting that the proposed algorithm creates children nodes of a parent node in breadth-first order, but traverses these children nodes in depth-first order. This is also evident from Figs. 3A and 3B. All the children nodes of the root node are created in breadth-first order, as shown in Fig. 3A. Similarly, all the children nodes of the node $S$ are also created in breadth-first order in Fig. 3B.

The rectangular nodes start to generate when the genFItems call itself recursively for the first time. This time, the nodes created have their support approximated in the previous call to genFItems, which was made from probDFAlgo. However, the rectangular children nodes of the round node $SW$, shown in Fig. 3C, are all infrequent, and the first recursive call to genFItems returns without making any addition to the search space tree.

The second recursive call to the genFItems routine in Fig. 3D is made for the round node SX. This call creates two rectangular children, $SXY$ and $SXZ$. Both of these children are frequent. Figure 3E shows that the third recursive call creates the rectangular node $SXYZ$. The diagram of the complete search space is shown in Fig. 3F.

The support calculated using the PSPM is probabilistic; therefore, these values are continuous. For example, to calculate the probabilistic support for the itemset $\left\{S,W,Y\right\}$ using Eq. (8), the following calculations are used:

$$support\left(SWY\right)={\displaystyle \frac{\left|S\cap W\right|\times \left|S\cap Y\right|\times \left|W\cap Y\right|\times \left|U\right|}{\left|S\right|\times \left|W\right|\times \left|Y\right|}}$$

$$support\left(FDB\right)={\displaystyle \frac{2\times 3\times 4\times 10}{4\times 6\times 8}=\mathrm{1.25.}}$$

Similarly, to calculate the support of $SXYZ$, put $Pref=S$, $A=X$, $B=Y$ and $C=Z$ in Eq. (9), and the answer is 2.17 (the more precise answer is from ProbDF).

$$\left|S\cap X\cap Y\cap Z\right|={\displaystyle \frac{\left|S\cap X\cap Y\right|\times \left|S\cap X\cap Z\right|\times \left|S\cap Y\cap Z\right|\times \left|S\right|}{\left|S\cap X\right|\times \left|S\cap Y\right|\times \left|S\cap Z\right|}}$$

$$support\left(SXYZ\right)={\displaystyle \frac{2.01\times 3.33\times 2.91\times 4}{3\times 3\times 4}=2.164\approx \mathrm{2.17.}}$$

Complexity analysis of the ProbDF

The total runtime of the ProbDF algorithm comprises two distinct parts: the time required for the support calculation of frequent itemsets of size 1 and 2, and the time required to generate and evaluate candidate itemsets belong to the level 3 and beyond. Let $n$ represent the total number of frequent items and $T$ represent the total number of transactions in a given dataset. In the worst-case scenario, the support calculation for frequent itemsets of size 1 and 2 assumes each frequent item is present in every transaction, resulting in a time complexity of $O\left(n\times T\right)$. Generating all possible candidate itemsets is a combinatorial problem that involves generating $2^n$ subsets of the set of $n$ frequent items, leading to a time complexity of $O\left(2^n\right)$. Therefore, the total runtime of the ProbDF algorithm in the worst case is $O\left(n\times T\right)+O\left(2^n\right)$. When considering the average runtime complexity, replacing $n$ with the Average Transaction Length ( $ATL$), we obtain $O\left(ATL\times T\right)+O\left(2^n\right)$.

The space complexity of ProbDF can be determined by examining the example presented in Fig. 3. The space required for the vectors, storing child itemsets support, at level 1 is ${\displaystyle \frac{\left(n\right)\left(n-1\right)}{2}}$, at level 2 is ${\displaystyle \frac{\left(n-1\right)\left(n-2\right)}{2}}$, and so on. For the second-to-last level, it is 1, and for the last level, it is 0. Let $\phi$ represent the height of the frequent itemset tree (i.e., the length of the longest frequent itemset). Then the space complexity is $\phi \left[{\displaystyle \frac{\left(n\right)\left(n-1\right)}{2}+{\displaystyle \frac{\left(n-1\right)\left(n-2\right)}{2}+\dots +1+0}}\right]+\phi$, which can be simplified to $O\left(\phi n^2+\phi \right)$.

Time efficiency of ProbDF

In Fig. 4A , the total runtime of the seven benchmark algorithms: NegFIN, dFIN, PrePost+, FP-Growth, HashEclat, RSB-PFI, and ProbBF, has been compared against the run time of the ProbDF algorithm. In this comparison, these algorithms have been applied to five benchmark datasets: Chess, Connect, Accidents, Pumsb, and Pumsb_star, and the results are shown in Fig. 4A.

As previously mentioned, the most processing-intensive operation is support discovery in frequent itemset mining. This is because it requires to examine all the relevant transactional data to be able to determine support of an itemset. Since the ProbDF does not use any transactional data in the support finding operation, this fact makes it the most efficient algorithm. This efficiency is visible in Fig. 4A across all the benchmark datasets and against all seven state-of-the-art algorithms. As stated earlier, all these algorithms in this comparison are configured in such a way that they discard frequent itemsets soon after they are generated. That is, the frequent itemsets are neither stored in memory nor written to disk. The reason for this setting is to strictly restrict the comparison to the very basic memory and computational requirements of the algorithms. We believe that the computational gains of algorithms are diluted by including the writing time (the time it takes to write the frequent itemsets to disk) in the total time the algorithms take to mine the datasets. Similarly, storing frequent itemsets in memory will also thin-out memory savings by the algorithms.

The ProbBF algorithm, as state earlier, has been recoded with queue data structure. However, the unique requirements of Eqs. (8) and (9) enforce that this queue data structure holds three recently traversed tree levels in memory. This contrasts with conventional breadth-first algorithms, which hold only one level in memory. However, the memory footprint of the Java-coded ProbBF is significantly smaller than that of the original version, as it utilizes a set enumeration tree to store all discovered frequent itemsets. Nevertheless, the Java-coded ProbBF still exhibits slower performance than NegFIN on the Connect dataset, which is notably dense. This discrepancy arises because NegFIN consumes lower memory than ProbBF when it refrains from storing any frequent itemsets in memory. Additionally, ProbBF is further disadvantaged by its lack of utilization of the promotion pruning concept, a feature that NegFIN employs effectively.

The other two approximate algorithms, HashEclat and RSB-PFI, perform poorly on dense datasets. As reported in Sadeequllah et al. (2024), their lack of effective compression of transaction data makes them less suitable options for dense datasets, i.e., these algorithms suit sparse data. On the other hand, ProbDF outperforms all seven algorithms by significant margins due to its simple computations with zero transaction data, efficient depth-first-based search strategy, and effective promotion pruning application.

Space efficiency of probDF

Since ProbBF and ProbDF algorithms do not calculate support from the transactional data for frequent itemsets of size three or more (i.e., $k$ ≥ 3), they delete all transaction data after determining all frequent 2-itemsets. This characteristic results in very low memory requirements for these two algorithms. However, ProbBF consumes more memory than ProbDF because its queue data structure stores the three most recently traversed levels of the search space tree. On the other hand, ProbDF’s newly proposed efficient depth-first search strategy stores partial results of only a small number of candidate frequent itemsets currently under consideration. Moreover, the application of promotion pruning further reduces the memory requirements of this algorithm by avoiding the storage and exploration of some branches of the search space. In conclusion, the combination of zero transaction data, memory-efficient depth-first search strategy, and promotion pruning renders ProbDF more memory-efficient than all other state-of-the-art FIM algorithms, as shown in Fig. 4B. ProbDF consumes the lowest memory on all benchmark datasets when compared to all competing FIM algorithms.

Analysis of the real frequent itemsets generation

Since ProbDF is a probabilistic algorithm, the frequent itemsets it generates could be classified as true positive, false negative and false positive. In this subsection, we analyze the strength of the ProbDF algorithm to generate the wanted true positive frequent itemsets, the unwanted false positive frequent itemsets and the missed frequent patterns (i.e., the false negative frequent itemsets). Let us call a frequent itemset as real frequent itemset (i.e., true positive) if its support count is above the min_supp threshold when calculated from the transactional data. Also, call the frequent itemset the ProbDF generates using PSPM be a probabilistic frequent itemset. This analysis has been done on all five selected benchmark datasets. Every dataset has been evaluated for a specific support threshold value, written in parenthesis along the dataset name in the diagrams. For example, in Fig. 5A, the ProbDF has been run on the Chess dataset for a support threshold of 0.4. This figure shows strength of the ProbDF to generate true positive frequent itemsets. For example, the red bar shows the real frequent itemsets for the chess dataset for the support value of 0.4 are 6,438,989. From these, the ProbDF has successfully generated 6,398,631, which are 99.4% of real frequent itemsets. These true positive frequent itesets for the Chess dataset are shown with blue bar in the figure. The remaining real frequent itemsets, which stands at 0.6%, have also been generated by ProbDF, but with one item missing from these itemsets. These itemsets have been shown with the brown bar along with the label 0.6%. The results of the other datasets are not very different than that of the Chess except the Connect. The results for the Connect dataset are unique in that non of the real frequent itemset has been missed by the ProbDF algorithm. This figure clearly shows that the true positive frequent itemsets generated by ProbDF are very close in numbers to the real frequent itemsets. In other words, ProbDF successfully generates majority of the frequent itemsets.

The analysis of the false positive frequent itemsets generated by the ProbDF is presented in Fig. 5B. In this figure, the blue bar represents the total real frequent itemsets for a given dataset. Similarly, the brown bar represents the total false positive frequent itemsets generated by the ProbDF for the given dataset and the mentioned support. These false positive frequent itemset are also represented by the green bar but with different metric, i.e., in percentage. The green bar shows what percent the false positive frequent itemsets are of the real frequent itemsets. This percentage value is written on top of the green bar. The blue bar and the brown bar are also labeled with the number of itemsets they represent. The lowest number of false positive itemsets is recorded for the Connect dataset, while the highest number of false positive itemsets is shown for the Pumsb_star dataset. A reason for this is the density of the database. As mentioned earlier, the Connect dataset is very dense, but the PUMSB_star database is low in density, as this dataset is derived from the PUMSB dataset by deleting all those frequent items whose support is 80% or above. These results clearly indicate that the more the dataset is dense and the support threshold is high, the PSPM produces better results.

In another figure, in Fig. 6A, all of the false positive frequent itemsets generated by the ProbDF have been shown with the real support. This figure also shows the average real support of these false positive frequent itemsets. Since these itemsets are not frequent, their support is less than the minimum support threshold (i.e., minimum support count in the figure). The average support of the Chess, Connect, Pumsb, Pumsb_star, and Accidents datasets (which are mined for the support threshold of 0.5, 0.7, 0.7, 0.4, and 0.4, respectively) are 0.491, 0.699, 0.692, 0.391, and 0.39, respectively. This figure clearly indicates that these false positive frequent itemsets are closely located on the outer sphere of the frequent itemset boundary.

Lastly, we apply two widely used measures in the data mining literature, recall and precision, to evaluate the effectiveness of the ProbDF algorithm in successfully approximating all frequent itemsets. Recall represents the percentage of all true positive frequent itemsets that are successfully predicted by ProbDF, while precision represents the percentage of probabilistic frequent itemsets generated by ProbDF that are true positive. This analysis is depicted in Fig. 6B for all four approximate algorithms compared in this study.

Two observations can be drawn from the results shown in Fig. 6B. Firstly, there are no significant differences in the quality of the results of ProbDF and ProbBF because they use the same PSPM model for their predictions. Secondly, the results of this figure closely align with those reported in Sadeequllah et al. (2024). This ensures that the quality of the predictions of ProbDF is the same as that of ProbBF, albeit with more time and memory efficiency.

Discussion of ProbDF algorithm outcomes and implications

The above findings reinforce the observations made in Sadeequllah et al. (2024). It was noted in Sadeequllah et al. (2024) that the PSPM technique produces accurate predictions, especially when the dataset is dense and the support threshold is high. This fact is further corroborated by the results shown in Figs. 5A and 5B, which depict the outcomes of mining the Connect and PUMSB_star datasets in this study. The Connect dataset, being highly dense and mined at a support of 0.7, exhibits significantly superior results compared to the less dense PUMSB_star dataset, which is mined at a lower support of 0.4.

Additionally, Fig. 6A illustrates that false negative itemsets have probabilistic support just below the minimum support threshold. This observation aligns with the results in Fig. 5A, where most false negative frequent itemsets are included in the probabilistic frequent itemset but with one item missing.

Moreover, the results shown in this section clearly indicate that the quality of ProbDF’s output deteriorates as the support threshold values are lowered. This decline is due to the scalability of the proposed methodology. As the size of the frequent itemsets increases down the search space, the error compounds, and the possibility of false outcomes rises. This compound error is also significantly influenced by the density of the dataset.

Figure 7 displays the densities of different datasets by dividing the average transaction length (after deleting the infrequent items) by the total number of frequent items. This figure shows that the Connect dataset has the highest density, while PUMSB_star has the lowest. Various studies (Burdick et al., 2005; Gouda & Zaki, 2005) report that the Average Frequent Itemset Size (AFIS) for the Connect dataset is double that of the AFIS for the PUMSB dataset at the same support threshold values. The PUMSB and Chess datasets have nearly identical AFIS, while the AFIS of the PUMSB_star dataset falls between those of PUMSB and Connect.

However, Figs. 5A, 5B, and 6B show that the quality of ProbDF’s output is highest for the Connect dataset, followed by somewhat identical performances on Chess and PUMSB datasets, while the quality for PUMSB_star is the lowest. This is attributed to the highest density of the Connect dataset, followed by Chess, PUMSB, and PUMSB_star. These findings highlight the significant role that dataset density plays in the successful application of the ProbDF algorithm. Additionally, the results do not indicate that ProbDF is affected by the number of frequent items or the number of frequent itemsets in a dataset.

To summarize the above discussion, the ProbDF algorithm is most effective with dense datasets, yielding higher-quality output when relatively high support threshold values are used. Additionally, ProbDF is valuable in FIM domains where some loss of quality is acceptable in exchange for increased efficiency. This is particularly advantageous in real-time environments where fast response times are prioritized over the highest quality responses. Based on the findings in this section, it is reasonable to conclude that ProbDF offers high runtime and memory efficiency with only a marginal loss in quality.