A novel trace-based sampling method for conformance checking

Process mining

PM is a research area that intersects data science and process science techniques to perform model discovery, conformance checking, and enhancement tasks of business processes by extracting knowledge from event logs. It is focused on studying and improving operational processes, i.e., processes requiring the repeated execution of activities to deliver products or services (van der Aalst, 2022). PM is supported by two main components: business process models and event logs.

A business process model is used to visualize and understand the behavior of a business process. It can be represented in various notation languages (Petri Net, BPMN, workflow, etc). A BPMN model of a business process can be formally represented as a directed connected graph $M=(N,E_m)$, where N corresponds to a set of nodes and $E_m$ is the set of directed edges that connect those nodes. N represents various nodes in the graph, generally associated with tasks in the business process or elements to control the flow of tasks according to the business process logic. N includes nodes in the tuple $\{i,o,T,G^{+},G^{\ast },G^o\}$, where $i$ is the node representing the initial event or starting activity in the process, $o$ is the end event or ending activity in the process, T is the set of all the tasks executed in the process. $G^{+}$ is the set of nodes representing AND gates. The outgoing edges of an AND-gate node point to tasks that must be executed in parallel. $G^{\ast }$ is the set of nodes representing XOR gates. Just one of the tasks pointed by the outgoing edges of an XOR-gate node is executed (depending on a given condition). $G^o$ is the set of nodes representing OR gates. Any or even more than one task pointed by the outgoing edges of an OR-gate node could be executed. An arc in M of the form $(a;b)$ implies that if $a,b$ $\in$ N, $b$ is a task executed after the execution of $a$. Figure 1 shows an example of a process model in health using the BPMN notation previously described. The process begins with the activity Register, followed by the parallel activities Collect symptoms, Triage, and Record vital signs (tasks executed in any order), followed by the activity Update medical record and the activities Take blood test and Check circulation, where only one of these last two activities can be executed. In the process model, the initial event (depicted by the circle to the left of Fig. 1) is triggered by the patient upon arrival. It ends with a final event (depicted by the circle to the right of Fig. 1). During the execution of the healthcare process, different actors are involved, such as healthcare personnel (physicians, nurses, assistants, etc.) and patients performing one or more activities. Business process models can be created by experts (documented model) or discovered from the event log (discovered model). Process model discovery is the most popular task in PM.

Event logs are created from the data collected during the execution of business processes activities, in the context of process instances (or cases). The execution of an activity in a given process case is referred to as an event $e_j$. The sequence of all the events for a specific process case $i$ is also referred to as a trace $t_i=e_1,e_2,e_3,...,e_m$. An event is associated with an activity in the business process. Thus, multiple traces can contain the same sub-sequence of activities, yet, since events are unique, each trace itself contains different events. Therefore, activities (associated to events) may appear more than once in a trace and in the entire log (Marin-Castro & Tello, 2021). Let A be the set of activities in the business process and $M(A)$ a multiset over A, where elements of A can appear multiple times. Formally, an event log L can be defined as a multiset of sequences over A, i.e., L $\in M(A^{\ast })$. Table 1 shows an example of an event log associated with the implementation and execution of the documented process model in Fig. 1. In this example, three cases of the process are shown (three traces) {<Register, Triage, Collect symptoms, Update record>, <Register, Triage, Update record>, <Register, Collect symptoms>}. In the example, activities associated to the events are shown in each trace.

Conformance checking

The goal of conformance checking is to relate the modeled behavior of a process, either in a documented or discovered model, to the recorded one in an event log. The general idea is to compare and analyze observed instances of a process in the presence of a model, either discovered or manually constructed (Carmona et al., 2018).

Conformance checking allows knowing some of the following important aspects of the business process:

• –Is the process being executed as it is documented in a model?

• –Is the model of a process still up-to-date?

• –Is compliance with standards and regulations maintained in the execution of process activities?

• –What is the level of flexibility allowed in the execution of the process?

Conformance checking may expose undesirable deviations in a business process and, thus, guide optimizations and improvements for the process. However, deviations could be desirable and intentionally generated by actors in the process to improve execution paths, not formerly foreseen at design time. Thus, while conformance-checking is a useful tool for decision-making, alignment is one of the most common conformance-checking techniques to achieve that goal.

Alignment is defined as a sequence of moves, each relating an event in the trace to an activity in the model. According to the cost function, an alignment with the lowest cost is optimal. Unfortunately, the problem of finding the optimal alignment is NP-hard (de Leoni & van der Aalst, 2013). This means that when the process is large, it is intractable to determine the optimal alignment, or if the event log is large, it may not be possible to reach or obtain the conformance value. That is why the motivation of this work is to propose an alternative method that addresses the alignment problem using a representative subset of the event log to estimate an approximate but confident conformance value.

In an alignment technique, an individual mapping of an event $e_j$ (in a trace $t_i\in L$) to an activity $a_k$ (in a model M) is referred to as a move and denoted by $\epsilon$. There are four types of moves:

• –Synchronous move (the move in both the log and the model): $e_j$ in any $t_i$ trace can be mapped to the occurrence of an enabled activity $a_k$ in M. This move can be expressed as the mapping ( $e_j$, $a_k$).

• –Move on model ( $mov{e}_M$): The occurrence of an enabled activity $a_k$ in M cannot be mapped to any event given the flow implied by a trace $t_i$. In this case, the behavior in M would not be observed in the trace. This move is denoted as a mapping $(\gg ,a_k)$. Here, the symbol $\gg$ denotes the absence of event $e_j$ that should correspond to $a_k$ in M.

• –Move on log ( $mov{e}_L$): Opposite to the previous case, an event $e_j$ in a trace $t_i$ cannot be mapped to any enabled activity in M. This move is expressed by the mapping $(e_i,\gg )$. Now, $\gg$ denotes the absence of activity $a_k$ in M that should correspond to $e_j$ in $t_i$.

• –Illegal move ( $mov{e}_I$): This move occurs when $e_j$ belonging to $t_i$ cannot be mapped in any way to an activity $a_k$ in M.

As an example, consider Fig. 2 showing three possible alignments of the trace $<a,b,c,e,f>$ in the BPMN process model M in Fig. 1. In the three alignments, the mapping ( $a$, $a$) is a synchronous move and refers to the first event in the trace corresponding to the first internal activity in the model M. While the second move in ${\gamma }_1$ and ${\gamma }_3$ is a synchronous move, it is a log move in ${\gamma }_2$. In the example, each alignment has the same number of move types, but not the same moves: four synchronous moves, two log moves, and one model move. As shown in the previous example, many possible alignments may exist between a process model and an event log. Let $A_{LM}$ be the set of legal moves implied by the alignment technique over the model M associated with the set of traces $\in$ L. An alignment $\gamma$ is a subset $\{{\epsilon }_1,{\epsilon }_2,...,{\epsilon }_k\}$ of $A_{LM}$, and corresponds to the alignment for a specific trace in L, for example, any of the ones presented in Fig. 2.

Let $\delta$ be the cost function $\delta :A_{LM}\to \mathbb{N}$ that assigns costs to legal moves in $A_{LM}$. Thus, the cost of an alignment $\gamma$ $\in$ $A_{LM}$ can be computed as $C_{\gamma }={\sum }_{i=1}^k\delta ({\epsilon }_i),{\epsilon }_i\in \gamma$ (Munoz-Gama, 2014). The costs may depend on the importance given to the activities within the process, i.e., some activities may affect the execution of the process more than others, so their omission or change of order during execution may cause severe problems. Conversely, other activities are more tolerable. Moreover, the severity assumed for log and model moves may be different, depending on the level of affectation. A standard cost function ${\delta }_S$ assigns costs to moves as follows:

• –A synchronous move has cost 0: ${\delta }_S(e_j,a_k)=0$.

• –A log move has cost 1: ${\delta }_S(e_j,>>)=1$.

• –A model move from a visible task has cost 1: ${\delta }_S(>>,a_k)=1$.

• –A model move from an invisible task (task without label) has cost 0: ${\delta }_S(>>,a_k)=0$.

• –An illegal move always has cost 0.

Consider ${\delta }_S$ for the alignments in Fig. 2. For example, for ${\gamma }_1$, its cost is ${\delta }_S({\gamma }_1)={\delta }_S((a,a))+{\delta }_S((b,b))+{\delta }_S((c,c))+{\delta }_S((>>,d))+{\delta }_S((e,e))+{\delta }_S((f,>>))+{\delta }_S((>>,g))=0+0+0+1+0+1+1=3$. Since the move types are the same in the three alignments, the cost will also be the same in the three cases.

Analysis of the event log

In the first stage, the event log is received as input for further processing to extract its features such as #events, #activities, and data associated to traces. The cardinality $|L|$ is used to determine how to proceed with the log to compute the sample of traces in the next stage. In case $|L|$ is greater than a given threshold (minTraces), it could be worthwhile to invest some effort in selecting an instance subset of the event log to reduce the workload in the conformance checking task. If this scenario holds true, we calculate the dispersion level ( $dispDP$) of L to assess the feasibility of employing a representative trace sample for the fitness computation. If the dispersion level does not exceed a specified threshold ( $\alpha$), we proceed to obtain the sample of traces in Stage 2.

Stage 1 and stage 2 of Fig. 3 are reflected in Algorithm 1. The data structure that models the event log L allows obtaining associated data such as the names of activities in the event log, total number of events, size of a trace $t_i$ $\in$ L, frequency of occurrence of a given activity in a trace $a_i$ or in all the event log, total number of activities in the event log, among others.

In order for our method to be applied, it is checked that the cardinality of the input event log is greater or equal to minTraces. This threshold is defined at implementation time. Usually, event logs of size average to large have an amount of cases or traces greater than minTraces to select a sample of traces for the conformance check task. Therefore, if a log with the described characteristics is received, the level of dispersion $dispDP$ of traces in L is calculated (lines 4–20 in Algorithm 1). We define dispDP as a metric that allows measuring the variability or spread of traces (sequences of events) within an event log. An event log with low dispersion and a large number of traces suggests uniformity among traces, allowing for simpler trace analysis through a sampling. Conversely, an event log with high dispersion has a wide variety of execution instances whose log behaviour may not be retrieved by simple trace sampling. When dealing with an event log with low dispersion, a subset of traces reduces the computational load, enabling faster processing while still yielding meaningful insights about the overall process. In the context of process mining and for this work, a data dispersion measure was used to identify the distribution of events in the traces. This was done because the content or characteristics of the event logs are not known in advance. Dispersion is a statistical measure widely used in natural language processing for corpus linguistics. In this work, we are using the dispersion measure for the first time in the event log analysis task of the proposed method. For this, a series of basic statistical measures, their characteristics and their manual calculation were studied. Derived from this study, the normalized version of the deviation of proportions (DP) dispersion measure (Gries, 2020) was selected and used. The DP dispersion metric can handle differently sized event logs, and it can immediately be applied to other kinds of data/scenarios such as activities co-occurrence frequencies in event logs. The DP measure can distinguish distributions that other measures fail to distinguish, measures do not have too high sensitivity unlike some other metrics, and it does not output extremely high values when low expected frequencies come into play.

DP for a given activity $j$ in a trace $i$ is calculated according to Eq. (1) as a value in the range [0,1]. A value closer to 1 indicates a higher dispersion in the event log. In that equation, $f$ is the overall frequency of an event in the event log; $V_{j,k}$ (computed at line 10 in Algorithm 1) represents the relation of the frequency of activity $j$ in the trace $k$; and $S_i$ (computed at line 6 in Algorithm 1) represents the relation of total events in the trace $t_i$ to the total events in the log.

(1) $$DPActi{v}_j=0.5\times \sum _{k=1}^n|V_{j,k}-S_i|.$$

Representative sample of traces

In this stage, we compute a representative sample (traceSample) from the complete set of traces. This sample serves as the input traces set for the conformance checking task in stage 3. Once the alignment value of a trace is determined, no additional time is needed to calculate the alignment of its similar traces. However, our approach aims to capture the behavior of the entire event log, especially focusing on traces with lower similarity.

According with our method, the sample of traces could be obtained in one of two forms: (1) by random sampling or (2) by selecting the full set of traces. The first case is when $(n=|L|$) $>minTraces$ and $dispDP\le \alpha$. The second one is when ( $n\le minTraces$) or $dispDP>\alpha$.

Conformance metric

Stage 3 of the proposed method uses a fitness function to calculate the conformance value between the process model M and the representative sample of traces L^′. The fitness function is shown in Eq. (3). It calculates the percentage of the sample’s behavior replayable by the process model, i.e., it determines the degree of correspondence or flow of control between a process model and its trace sample. Each time a trace in the event log achieves optimal alignment with the model, the alignment cost is reduced so that the set of traces contributes to determining the full alignment cost. A fitness value equal to 1 indicates that all the behaviors in the event log are described by the process model, or in other words, all traces in the event log are replayed in the process model.

(3) $$fitness(L,M)=1-\frac{fcost(L,M)}{mov{e}_L(L)+|L|\times mov{e}_M(M)}.$$

In Eq. (3), $fcost(L,M)$ is a standard cost function as described in “Conformance Checking”, and $mov{e}_L$ and $mov{e}_M$ correspond to the number of moves in the log and model, respectively.

Data

The event logs used for experimentation are: Sepsis Cases (Mannhardt, 2016), regarding the sepsis medical condition of a hospital; Hospital Billing (Mannhardt, 2017), containing billing data for medical services; Hospital log (or BPIC2011) (van Dongen, 2011), with cases of patients of gynecology department; Road (de Leoni & Mannhardt, 2015), a log of an information system managing road traffic fines; and BPIC2012 (van Dongen, 2012), related to loan or overdraft applications at a Dutch financial institute. While the first three event logs are representative in the healthcare domain reported in the literature, the two last are considered for a qualitative comparison against related works. Although the datasets considered for experimentation are of different domains, all of them are suceptible to present different levels of dispersion in the execution of their activities. Besides, the datasets are event logs with a large number of cases, suitable for assessing our proposal under conditions that favor trace selection. The description of each log is presented in Table 2.

Setup

The parameters selected for the dispersion threshold $\alpha$ and minimum number of traces $minTraces$ used in Algorithm 1 were recommended according to the following criteria. The dispersion level dispDP is in the range [0, 1], where a value close to 0 indicates that the activities are distributed across the event log as expected. A value close to 1 indicates the opposite (activities distributed in the event log unexpectedly). Therefore, we consider that an event log has a high dispersion if dispDP is close to 1, medium dispersion if dispDP in the range [0.5, 0.7], and low dispersion if dispDP is below 0.5. In our conformance approximation method, it was recommended to apply the random probability sampling strategy on those event logs with a lower and medium level of dispersion, that is, $\alpha \le 0.7$.

It is known that there is no exact way to classify event logs (or databases) into large, medium, or small size. However, since the task of conformance checking requires a high computational cost when working with large logs, the dispersion level is only computed in those logs of medium to large size, that in this work are those logs with more than 100 traces. Othercase, small logs (with less than 100 traces) are no considered in our conformance checking strategy.

Experiment 1. Calculation of the event log dispersion level

In the first part of the experimental evaluation, a study of the dispersion level of the event log data was carried out considering the three logs described in the previous section Data.

Table 3 presents in detail the dispersion level obtained for each activity recorded in the event log. The dispersion level behaves well for the activities with the highest frequencies, obtaining a low dispersion level. Column 2 lists the activities in the event log; column 3 presents the dispersion level for each activity; column 4 the occurrence frequency of activity in the event log; and finally, column 5 presents the dispersion type. Due to the large number of activities in the BPIC2011 event log, only 10 of these activities are presented.

Also, Table 4 shows the average dispersion level obtained by the three event logs, where BPIC2011 presents the highest level, possibly due to the wide variety of activities and the diversity in the size of its traces. The three logs present high and low dispersion levels according to the previously recommended classification; thus, the sample of traces was computed in all cases. Also, Table 4 shows the size of the trace sample obtained according to Eq. (2). In all cases, the subset of traces for the conformance checking task was much less the size of the entire event log: 80%.

To understand the practical application of the dispersion level in the sampling process, we conducted an analysis to explore its correlation with performance metrics, specifically, the accuracy loss in fitness and the increase in processing time. To achieve this, we constructed two synthesized event logs: one with 110 traces (Log110) and another with 200 traces (Log200). We intentionally introduced varying degrees of dispersion among the activities within each trace in both logs. This allowed us to investigate whether a correlation exists between the dispersion level and both fitness accuracy and processing time. Table 5 shows the results obtained from the experiment conducted. From the presented results, it is concluded that whenever the level of dispersion within the event log rises, both fitness accuracy and execution time are impacted. While the effect may occasionally be marginal, there exists a correlation between the dispersion metric and the accuracy of fitness as well as processing time.

Experiment 2: performance evaluation

Figure 4 shows the performance evaluation of the proposed method, expressed as the response time that was calculated as the sum of the timing for the different main processes: event log transformation TR, calculation of the dispersion level DIS, and the calculation of the size of the representative sample SAM. All these processes cover the data processing cycle, starting from the transformation of the raw event log until the traces sample subset is obtained. According to the obtained results (range from 0.09 s to over 500 s), it is hard to generalize the method’s behavior in the TR, DIS, and SAM processes. In two of the three event logs, TR was negligible if compared to DIS or SAM. However, that is not the case for the event log of BPIC2011, where the TR cost is almost half the total response time due to the number of activities considered by BCPI11, as well as the average size per trace, which is up to nine times larger compared with the size of the traces in Sepsis Cases and Hospital Billing. Note that the Sepsis Cases and BPIC2011 have a similar number of traces but not similar trace length, dispersion level, and amount of activities, which translates into higher processing times in the three processes TR, DIS and SAM, with considerably larger distance between these two event logs. In this same test, only the response time to calculate the conformance value was obtained for both cases by using the complete event log and when using only the sampled event log.

Figure 5 shows the time for computing the conformance value for the three event logs. It can be noted that the greatest time corresponds to BPIC2011. Despite not being the log with the largest number of cases compared to Hospital Billing, the BPIC2011 log is characterized by containing very long traces, and the dispersion level is higher than the other two event logs, which makes the conformance checking process much slower compared to the other two logs. However, Fig. 5 reveals that the time is considerably less when subsets of samples from the original event logs are used. In all cases, there are notable time savings when using sampled event logs, from 75% with the Sepsis Cases sampled log to 98% with the BPCIP2011 sampled log.

Experiment 3: measuring the conformance value

In this third experiment, the conformance value was assessed considering the alignments computed against every trace in the log as baseline. While improving the processing time of the conformance-checking task is important, achieving conformance values similar to those obtained with the original event log is essential to ensure that there is no high loss of process behavior. For this purpose, the conformance value between process models and the complete and sampled event logs was analyzed.

Table 6 shows the results obtained from conformance assessment using the fitness metric (Eq. (3)) under the two scenarios described previously. Together with fitness, we also provide in Table 6 results for precision and generalization to show that the use of a sampled trace set, selected as proposed in our article, does not greatly affect these common metrics used in process mining. On the one hand, precision estimates the extent of behavior allowed by the process model that is not observed in the event log. On the other hand, generalization estimates the extent of behavior allowed by the process model that is not observed in the event log. Both, precision and generalization were obtained from the ProM tool. Columns 3 and 4 show the results of the evaluation obtained using the complete and sampled event log built according to the proposed trace-based sampling method. The last column presents the loss in each metric for the case when using the sample subset of traces instead of the complete event log. According to these results, this loss, particularly in fitness, is small in most cases. For example, for the BPIC2011 log, the loss is 7.4%. A possible cause of this situation is the level of dispersion and the number of activities considered by BPIC2011. For event logs characterized by a high level of dispersion, such as BPIC2011, it is advisable to utilize the entire set of traces rather than applying sampling to mitigate the risk of experiencing substantial fitness loss. In the case of the precision metric, the loss is most notable in both Hospital Billing and BPIC2011 logs, which results in the process model admitting behavior not seen in the event log.

Also, this third experiment compared the conformance level obtained for the three event logs under three different configurations. This is shown in Fig. 6, where the light blue bars represent the conformance value between the process model (discovered from the complete log) and the complete log (case referred to as CCL); the orange bars show the conformance value between the process model (discovered with the sampled log) and the sampled log (case referred as SSL); finally the grey bars show the conformance level between the process model (built with the complete log) and the sampled log (case referred as CSL). This is to verify if there is a loss or gain in the level of conformance obtained by reducing behavior in the process model discovered with a sampled event log. At the same time, to analyze how that difference is concerning the three configurations to determine which of these configurations has a better conformance level.

As shown in Fig. 6, the fitness obtained for the Sepsis Cases and Hospital Billing logs in the CSL configuration is better than the one obtained in CCL or SSL. These cases demonstrate that the method for selecting a traces subset retains the process’s behavior and better fits the model. In the case of the BPCI2011 log, the fitness value is maintained for both SSL and CSL configurations. However, there is a noted difference between the latter two configurations and CCL due to the particular characteristics in the event log, such as a high level of dispersion and many activities. In all cases, a small loss in fitness was experienced because the process model was obtained with a sample of the event log.