Accelerating covering array generation by combinatorial join for industry scale software testing

Combinatorial interaction testing

Combinatorial Interaction Testing (CIT) technique generates a test suite that contains all the possible combinations of values among any t parameters for a system under test. A test suite generated by a CIT tool is called a coveringarray. It is denoted as CA(N; t, k, v), where N is the number of rows, t is called testing strength, k is the number of columns (i.e., parameters), and v is the number of possible values for each parameter.(here we assume each parameter has the same number of possible values) k and v are called degree and order respectively (Kuhn, Kacker & Lei, 2013).

CIT is useful to shrink the full Cartesian product space of a set of parameters, which becomes impractical for large-scale applications, into a reasonable test suite. The test suite generated by a CIT tool is called a covering array.

The most common type of covering array in CIT is pairwise (t = 2) in which all two-way combinations of parameter values are tested together in at least one test case. Numerous algorithms have been proposed to generate such artifacts (Nie & Leung, 2011; Anand et al., 2013), from greedy algorithm (e.g., AETG (Cohen et al., 1997a), IPOG (Lei et al., 2008, and PICT (Czerwonka, 2006)), simulated-annealing (Garvin, Cohen & Dwyer, 2011) to heuristic search-based technique (Shiba, Tsuchiya & Kikuno, 2004).

CIT has been applied to various applications including GUI testing, configuration-aware system testing (such as product line testing), and unit testing. A study in 2018 reported 40 commercial or open source tools have been developed to generate CIT test suites (Czerwonka, 2018).

The generation of a covering array has been extensively studied, to minimize the size of a covering array, to deal with constraints defined in a test model(Grindal, Offutt & Mellin, 2006; Wu et al., 2019), or to generate a covering array by extending an existing covering array (i.e. incremental generation, Kampel, Garn & Simos (2017); Kruse (2016); Zamansky et al. (2017); Ukai et al. (2019)), rather than from scratch.

Constraint support by existing tools

In a practical software system each parameter cannot be assigned independently. Instead, parameter values must be selected so that a certain set of conditions are satisfied. Such conditions are called constraints. For example, when we test a system equipped with web-based GUI, OS (Windows, Mac OS, Linux, etc.) and browser (Edge, Safari, Chrome, Firefox, etc.), OS and browser are parameters and their values specified in the parentheses are different settings that a user may access to the system. In a test case where Safari or Edge is chosen as the parameter browser, Linux cannot be assigned as an OS parameter. This is an example of a constraint. If a test case violating a constraint is introduced in a test suite, it will not cover the expected combinations of values, even those not related to the constraint, because this whole test case will not be valid. As a result, the combinatorial coverage of the whole test suite will be damaged. Specifically in our example, when we create a test case where Safari is chosen for Browser and Linux for OS, the test case is expected to cover valid value combinations for other parameters such as Font, Language, Timezone. Now the test case is violating a constraint about OS and browser and it makes the entire test case invalid. This means combinations for the other parameters (Font, Language, etc.) will not be executed unless they are accidentally covered by other test cases. Accidental coverage occurs much less frequently than one may expect because the CIT minimizes the number of tests cases to avoid repeating the same value combinations. Constraints are often denoted in a format of tuples that are forbidden to be present in the output covering array. For example, the constraint that Linux of OS cannot be tested together with Safari of browser is denoted as (OS_Linux, browser_Safari), where OS and browser are names of parameters and Linux and Safari are their values.

ACTS has a superior performance with respect to both generation speed of covering arrays and covering arrays size without constraints, based on a comparison between various tools conducted by Kuhn, Kacker & Lei (2013). For example, when ACTS generates a covering array of CA(2, 2, 100) with no constraint, it takes less than 1.0 [sec] and the size of the generated covering array is 14. Another popular tool, PICT can generate a covering array of CA(2, 2, 100) in less than 1.0 [sec] with 15 rows, but it shows quite unpractical performance when a complex constraint set is present (Czerwonka, 2016).

However, in terms of ability to define or describe complicated constraints and parameters (we call it flexibility), other tools (e.g., PICT and JCUnit) outperform ACTS. Flexibility of defining constraints is less researched than performance of generating covering arrays under constraints, but it is very important in practice. The effort to define constraints is necessary to model relationships between parameters and such a model sometimes becomes so complex that it requires a notation as powerful as a popular programming language, where products under testing are developed. On the other hand, introducing such a rich feature into the notation to describe constraints makes it difficult to implement an efficient covering array generator because constraint handling sometimes relies on an external SAT solver, which is not as powerful as a general purpose programming language such as Java.

In short, no single CIT tool provides superior performances for all requirements such as size, speed, and flexibility in constraint handling, simultaneously.

We next describe three tools studied in our research, ACTS, PICT, and JCUnit, with a focus on their different characteristics in defining constraints.

 
 
_______________________ 
                        < Parameters > 
                                                                                                                     < Parameter id =”2” name =” enum1 ” type =”1”> 
                                                                                                                     < values > 
                                                                                                                     < value > elem1 </ value > 
                                                                                                                     < value > elem2 </ value > 
                                                                                                                     </ values > 
                                                                                                                     < basechoices /> 
                                                                                                                     < invalidValues /> 
                                                                                                                     </ Parameter > 
                                                                                                                     < Parameter id =”3” name =” num1 ” type =”0”> 
                                                                                                                     < values > 
                                                                                                                     < value >0</ value > 
                                                                                                                     < value >100</ value > 
                                                                                                                     . . . 
                                                                                                                                                    < value >2000000000</ value > 
                                                                                                                     </ values > 
                                                                                                                     </ Parameter > 
                                                                                                                     < Parameter id =”4” name =” bool1 ” type =”2”> 
                                                                                                                     < values > 
                                                                                                                     < value > true </ value > 
                                                                                                                     < value > false </ value > 
                                                                                                                     </ values > 
                                                                                                                     </ Parameter > 
                                                                                                                     < Parameter id =”5” name =” range1 ” type =”0”> 
                                                                                                                     < values > 
                                                                                                                     < value >0</ value > 
                                                                                                                     < value >1</ value > 
                                                                                                                     < value >2</ value > 
                                                                                                                     < value >3</ value > 
                                                                                                                     </ values > 
                                                                                                                     </ Parameter > 
                                                                                                                     . . . 
                                                                                                                                                    </ Parameters > 
_____________________________________________________________________________________________________________________    

 
 
____________________________________________________________________________________________________________________ 
                                                                                                                     < Constraints > 
                                                                                                                     < Constraint text =” l01 & lt ;= l02 || l03 & lt ;= l04 
                                                                                                                                                    || l05 & lt ;= l06 || l07 & lt ;= l08 || l09 & lt ;= l02 ”> 
                                                                                                                     < Parameters > 
                                                                                                                     < Parameter name =” l01 ” /> 
                                                                                                                     < Parameter name =” l02 ” /> 
                                                                                                                     < Parameter name =” l03 ” /> 
                                                                                                                     < Parameter name =” l04 ” /> 
                                                                                                                     < Parameter name =” l05 ” /> 
                                                                                                                     < Parameter name =” l06 ” /> 
                                                                                                                     < Parameter name =” l07 ” /> 
                                                                                                                     < Parameter name =” l08 ” /> 
                                                                                                                     < Parameter name =” l09 ” /> 
                                                                                                                     </ Parameters > 
                                                                                                                     </ Constraint > 
                                                                                                                     </ Constraints > 
_____________________________________________________________________________________________________________________    

 
 
____________________________________________________________________________________________________________________ 
                                                                                                                    PLATFORM : x86 , ia64 , amd64 
                                                                                                                                                   CPUS : Single , Dual , Quad 
                                                                                                                                                   RAM : 128 MB , 1 GB , 4 GB , 64 GB 
                                                                                                                                                   HDD : SCSI , IDE 
                                                                                                                                                   OS : NT4 , Win2K , WinXP , Win2K3 
                                                                                                                                                   IE : 4.0 , 5.0 , 5.5 , 6.0 
____________________________________________________________________________________________________________________________________________________    

 
 
___________________________________________________________________________________________________________________________________________________ 
                                                                                                                                                   IF [ PLATFORM ] in {” ia64 ” , ” amd64 ”} THEN [ OS ] in {” WinXP ” , ” Win2K3 ”}; 
                                                                                                                                                   IF [ PLATFORM ] = ” x86 ” THEN [ RAM ] <> ”64 GB ”; 
____________________________________________________________________________________________________________________________________________________    

 
 
___________________________________________________________________________________________________________________________________________________ 
                                                                                                                                                   ! PLATFORM = ia64 && ! PLATFORM = amd64 || ( OS = WinXP || Win2K3 ) 
                                                                                                                                                   ! PLATFORM = x86 || ! RAM = 64 GB 
____________________________________________________________________________________________________________________________________________________    

 
 
___________________________________________________________________________________________________________________________________________________ 
                                                                                                                                                    @Condition ( constraint = true ) 
                                                                                                                                                    public boolean discriminantIsNonNegative ( 
                                                                                                                                                    @From (” a ”) int a , 
                                                                                                                                                    @From (” b ”) int b , 
                                                                                                                                                    @From (” c ”) int c ) { 
                                                                  return b ∗ b − 4 ∗ c ∗ a >= 0; 
                                                                                                                                                    } 
___________________________________________________________________    

 
 
__________________________________________________________________ 
                                                                  @ParameterSource 
                                                                                                                                                    public Simple . Factory < Integer > depositAmount () { 
                                                                  return Simple . Factory . of ( asList (100 , 200 , 300 , 400 , 500 , 600 , −1)); 
                                                                                                                                                    } 
 
                                                                  @ParameterSource 
                                                                                                                                                    public Regex . Factory < String > scenario () { 
                                                                  return Regex . Factory . of (” open deposit ( deposit | withdraw | transfer ){0 ,2} getBalance ”); 
                                                                                                                                                    } 
___________________________________________________________________    

Reuse covering arrays

Generating a covering array is an expensive task, especially when executed under complex constraints, a higher strength than two, and/or there are a number of parameters. Since a large software system can have a complex internal structure and hundreds or even more parameters, divide-and-conquer approach is desirable. If the time of covering array generation grows non-linearly along with the number of parameters n (e.g.,  n², n³), this approach may accelerate the overall generation because a set of parameters can be divided into multiple groups. Dividing into groups can prevent an explosive increase in the generation time for each group, even if there is overhead to recombine them into one .

To enable such an approach, a method to construct a new covering array reusing existing ones is necessary. However, such methods are not as well studied as methods to generate covering array from scratch (Kampel, Garn & Simos, 2017; Kruse, 2016; Zamansky et al., 2017; Ukai et al., 2019).

The most popular method for reusing a covering array is a feature called “seeding” (Cohen et al., 1997b). Seeding takes an existing covering array and parameters to be added as inputs. Hereafter, we refer to this method as incremental generation. This allows mandatory combinations to be specified for a tool, minimizing changes in the output. Minimizing changes is important because the output, which represents a test suite, sometimes contains fundamental parameters that are expensive to control such as OS or filesystem to be used in test execution. Popular tools for CIT such as ACTS (Kuhn, Kacker & Lei, 2008), PICT (Czerwonka, 2006), and JCUnit (Ukai 2007) can add parameters not presented in an initial covering array and generate an output as by assigning values to them so that the combinations between the values of the given parameters and the existing ones are covered. However, this limits reuse of only one covering array.

Another approach is to apply a CIT technique by setting each input covering array is a parameter whose rows are possible values (Zamansky et al., 2017). One drawback to this approach is that it makes the final array’s size larger than M × N, where M is the maximum array’s size in the input and N is the second maximum’s size. This results in an output with an impractical size for large-scale software product development.

As a third approach, in our previous work, we proposed an operation called combinatorial join (Ukai et al., 2019) to reuse covering arrays. Combinatorial join assumes that input arrays are already covering arrays and a new row in the output is created by connecting rows in the input arrays so that the entire output becomes a new covering array which has all the parameters to test. Ukai et al. (2019) presented an implementation of the combintorial join operation based on a covering array generation algorithm called IPOG (Lei et al., 2008). However, the implementation was impractically expensive in terms of time and memory usage when there are more than 100 parameters or strength t exceeds 2.

Variable strength covering array

A variable strength covering array (VSCA) is a covering array where the strength t can be different depending a set of parameters among all of them (Cohen et al., 2003). It is considered useful to apply VSCA for testing a system which consists of multiple components since some components are more critical than others in a large system. Methods to generate VSCA have been proposed in related work (Bansal et al., 2015; Wang & He, 2013).

As introduced later in ‘Weaken-product-based Combinatorial Join Technique’, our proposed combinatorial join operation can also generate a VSCA, because this approach guarantees to include all the rows in input arrays at least once, if one array has a higher strength than the other, the portion corresponding to the array will have the same strength as the input.

A running example

We present a running example of our proposed algorithm weaken-product based combinatorial join with a concrete example (Fig. 1) where both the input arrays’ and the output array’s strength are t = 2. In this example, the original LHS is a covering array that contains three parameters (i.e., OS_L, Lang_L, and TZ_L), each of which has two possible values Unix, Win, JA, EN, and EST, JST respectively. There is no constraint across LHS and RHS. Note that LHS and RHS can have different numbers of rows (i.e., different sizes) and columns as shown in the diagram (Fig. 1). The original RHS is also a covering array that contains three parameters which are OS_R, Lang_R, and TZ_R and they have the same possible values as the corresponding one in LHS. The goal of our algorithm (or method) is to combine them into one covering array that covers all the t-way combinations (in this example, t = 2) across the LHS and the RHS arrays without creating a new row neither in LHS nor RHS part.

First, the weaken operation, which shrinks the input covering array into another one with lower strength, is executed for both LHS and RHS (Step 1). The operation can have only one output. In general, the output arrays of this step in LHS will be covering arrays with strength t − 1, t − 2, …, 1, while the corresponding arrays from RHS will be 1, 2, …, t − 1. In this example, after this step, the output of LHS is only one covering array with strength 1 because the strength of the original LHS is t = 2, and the output of RHS is also one covering array whose strength is 1. Next, for each pair of output arrays of Step 1, a Cartesian Product is performed and the results are merged into one (Step 2). As it is seen in the figure, for each row in the output of Step 1 from LHS, every row in the output of Step 1 from RHS is connected. For instance, for a row (Unix, JA, JST) in LHS, every row in the output of the weaken operation for RHS (Unix, JA, JST), (Win, JA, EST), (Win, EN, JST) is associated.

In this step, rows in the output with exactly the same values for all parameters are removed. This removal is necessary when the weaken − product is performed for the strength higher than 2 because the Step 1 is repeated multiple times and it may generate duplicated rows in the output.

Then, the remaining rows in LHS and RHS that do not appear in the output of Step 2 are connected and included in the final output in (Step 3). For example, the row (Unix, EN, EST) in LHS and RHS is not found in the output of Step 2 and unless Step 3 is done to make up the missing tuples, not all the t-way combinations inside the LHS and RHS are ensured to be covered. Step 2 guarantees that t-way combinations of parameter values across LHS and RHS are covered. Step 3 guarantees t-way combinations of parameters inside LHS and RHS are covered. Therefore, the entire output becomes a covering array of strength t. Finally, the rows generated in Step 2 and Step 3 are merged into one array (Step 4).

Notation

Now we define some notations in order to formalize our proposed method “weaken-product based combinatorial join” in Method of “weaken-product based combinatorial join”. We first introduce a set of necessary functions before describing our proposed function, weaken_product(LHS, RHS, t) that builds a new covering array from two input arrays. The function takes three parameters, LHS, RHS, and t. The output of the function is an array containing all the factors held by the input arrays. LHS and RHS are arrays that do not have the same factors in common. In general, they are covering arrays of strength greater than t, although this condition is not mandatory. For simplicity, we assume that LHS and RHS do not have any constraints inside them. However, the proposed mechanism can handle those under constraints transparently. If the input has higher strength, it will be kept in the output, too, and if its rows do not violate given constraints, rows in output will also not violate the constraints. This is given as (2)${\displaystyle weaken\left(A,i\right)=A_w}$

where weaken is a function that returns a new array from input A. The output has the following features:

• It has all the factors in A and only those factors.

• It contains all the tuple of strength i that appear in A.

• It contains rows that appear in A and only those.

• Each row in the array is unique.

When output of the weaken(A, i) is constructed, depending on the order of selecting rows from A, the size of the output can be different. Our implementation chooses to select a row that contains the most key-value pairs that are not covered in the output so far.

In the case the input A is a covering array of strength i or greater, weaken(A, i) will be a covering array of strength i and its size can be smaller than A. This is expressed as (3)${\displaystyle \left|weaken\left(A,i\right)\right|\le \left|A\right|}$

factors is a function that returns a set of factors on which a given array is constructed. (4)${\displaystyle factors\left(A\right)=F}$

F is a set of all the factors that appear in an array A

project(A, f) is a function that returns an array created from an input array A and a set of factors f. (5)${\displaystyle project\left(A,f\right)=P}$

The returned array P satisfies the following characteristics.

• It has all the factors given by f only.

• For each row in P, a row in A, which contains the row, can be be found.

connect is a function that returns an array created from a couple of given arrays, L and R. (6)${\displaystyle connect\left(L,R\right)=C}$

The returned array satisfies following the characteristics.

• It has all the factors that appear in L and R.

• project(C, factors(L)) contains all the rows found in L and all the rows in it are contained by L.

• project(C, factors(R)) contains all the rows found in R and all the rows in it are contained by R.

• Each row in C has values for all the factors from L and R.

• Each row in the array is unique.

Since there is not a requirement for combinations of rows from A and B, |C| can be as small as max(L, R). (7)${\displaystyle set\left(A\right)=S}$

S is a set that contains all the identical rows in an array A.

Method of “weaken-product based combinatorial join”

Based on the formulae in ‘Notation’, the operation we propose weaken_product can be defined as follows. (8)${\displaystyle \begin{array}{c}\hfill WP=weaken\text{\_}product\left(LHS,RHS,t\right)\\ \hfill =\left[\bigcup _{i=1}^{t}weaken\left(LHS,i\right)\times weaken\left(RHS,t-i\right)\right]\cup connect\left(LH{S}_{unused},RH{S}_{unused}\right)\end{array}}$

where (9)${\displaystyle \begin{array}{c}\hfill LH{S}_{unused}=LHS\setminus project\left(W,factors\left(LHS\right)\right)\\ \hfill RH{S}_{unused}=RHS\setminus project\left(W,factors\left(RHS\right)\right)& \hfill \\ \hfill W=weaken\text{\_}product\left(LHS,RHS,i\right)\end{array}}$

Figure 2 illustrates the idea of the weaken_product function.

Next, we describe the characteristics of the output arrays generated by our proposed algorithm, in order to explain why we can use our algorithm to combine covering arrays generated under constraints. Given a set of parameters with their possible values, as well as a set of t − way tuples that is called “Forbidden tuples”, an array that covers all the possible t-way tuples but the forbidden ones is called a “constrained covering array” or CCA (Cohen, Dwyer & Shi, 2008). The set of forbidden tuples are determined by the constraints under which a covering array is generated for the system under test.

Suppose that LHS and RHS are constrained covering arrays generated under constraints with strength t. All rows in LHS are ensured to exist in WP and no new row is introduced according to Eqs. (2) and (8) ken,eq:weaken_product. This is also true for RHS. This leads to Theorem 1.

Theorem 1 (10)${\displaystyle set\left(project\left(WP,factors\left(LHS\right)\right)\right)=set\left(LHS\right)}$ (11)${\displaystyle set\left(project\left(WP,factors\left(RHS\right)\right)\right)=set\left(RHS\right)}$

We demonstrate that WP is a CCA generated under the constraints of LHS and RHS. From the precondition of the operation, there is no constraint across LHS and RHS. It is clear that there is no row that violates given constraints in WP. A tuple T ( |T| = t) that should be covered by WP, can be categorized into three.

• A tuple inside LHS (Eq. (10)).

• A tuple inside RHS (Eq. (11)).

• A tuple across LHS and RHS.

All the tuples that should be covered by WP inside LHS and RHS are found in the array (Theorem 1). In order to guarantee all the tuples across LHS and RHS are found in the WP, it is sufficient to include: (12)${\displaystyle weaken\left(LHS,i\right)\times weaken\left(RHS,t-i\right)}$

where 0 < i < t. Those are guaranteed to be in WP by the definition of the weaken − product operation defined as Eq. (8).

Thus, we can construct a new CCA from the existing CCA’s without inspecting into neither the semantics of the constraints nor the forbidden tuples defined for the input arrays. This allows users to employ an approach, where different CIT tools to construct input covering arrays and then combine them into one, later.

The same discussion holds for constructing VSCA, when input arrays are the covering arrays of the higher strength than t.

General definition of combinatorial join

We can generalize the operation we discussed in a way where our proposed method and Ukai et al. (2019) can be considered as implementations of one abstract operation based on the ideas introduced in ‘Notation’. This improves the approach in our last work. The characteristics that are desired for the output of the operation can be described as follows. (13)${\displaystyle set\left(project\left(combinatorial\text{\_}join\left(LHS,RHS,t\right),factors\left(LHS\right)\right)=set\left(LHS\right)\right)}$ (14)${\displaystyle set\left(project\left(combinatorial\text{\_}join\left(LHS,RHS,t\right),factors\left(RHS\right)\right)=set\left(RHS\right)\right)}$ (15)${\displaystyle \begin{array}{c}\hfill tuples\left(project\left(combinatorial\text{\_}join\left(LHS,RHS,t\right),t\right)\right)\supset \\ \hfill tuples\left(LHS\times RHS,t\right)\setminus tuples\left(LHS,t\right)\cup tuples\left(RHS,t\right)\end{array}}$

where tuples(A, t) is a function that returns a set of all the t-way tuples in an array A.

In this definition, note that any requirements are not placed on the input arrays. They do not need to be even any sort of covering arrays. These characteristics ensure that the operation does not introduce a new row that may violate constraints given to LHS or RHS and that it covers all the possible t-way tuples in and across LHS and RHS.

Research questions

In order to evaluate our technique from the aforementioned perspectives, we are going to answer the following research questions:

• RQ1: Can our weaken-product combinatorial join technique accelerate the existing CIT tools in covering array generation?

• RQ2: How are the sizes of covering arrays generated through our combinatorial join technique compared to the sizes of covering arrays generated without it?

• RQ3: Can our approach reuse test oracles?

• RQ4: How can our approach handle constraints with flexibility?

There is another approach that constructs a new covering array from existing ones (Zamansky et al., 2017). However it relies on converting an input array into a factor by reckoning each row in it as a level of the factor. This approach is not practical unless the number of factors are small. Due to the scalability issue, it is inapplicable to the experiment subjects used in our study. Hence, we are not going to compare our approach’s performance with their method but with that of ACTS.

Evaluation methodology

In this section, we describe how we conduct evaluation to answer each research question, and we illustrate how each research question relates to the covering array generation process in Fig. 3.

In order to answer RQ1, we measure the execution time of our algorithm including necessary preprocesses for the input data for a desired input model. The preprocess may contain a covering array generation since our algorithm does not generate a covering array but it takes two covering arrays as input. It will be compared with the execution time to generate a covering array using a conventional method for the same desired output model.

In order to generate covering arrays in our experiments, we need an external tool that executes the process and we chose ACTS for it. The reason why we chose ACTS is because it is not only widely used but also the fastest one among the tools available for us. We considered PICT as another choice, however it turned out to be too slow for our experiments because of its specification, where its covering array construction with constraint handling requires exponential time along with the number of factors (Czerwonka, 2016).

Similarly, the sizes of the generated covering arrays by the proposed method and conventional method are compared (RQ2).

When covering array generation is executed from scratch, the preprocess for the desired covering array model consists of two parts as illustrated in Fig. 4. One is to split the mode into LHS and RHS and the other is to generate covering arrays for them respectively. For splitting the model, we can think of some strategies. One is to divide the input into two groups each of which has the same number of factors.

Moreover, well-known covering array generation tools support a feature called “seeds“ or “incremental generation”, where an existing covering array is given as input whose rows are ensured to appear in output. This feature enables users to reuse test cases, test results, test oracles, etc. along with the input covering array. In this scenario (Fig. 5), the requirements for the final output (“Desired Covering Array Model” in the diagram) and base covering array for the conventional method are given as input. On the other hand, for our method, the factors to be added to the seeds are given separately (“RHS Model” in the diagram) and it is necessary to take into account the time to generate a covering array for it. However, the base covering array can be used as LHS without any preprocessing.

Our approach constructs a new row by selecting rows from input arrays instead of constructing it from scratch, so it has less options to optimize (minimize) the size of its output. As a result, our approach cannot generate a smaller output array than the conventional method (RQ2). In order to answer RQ2, we will compare the size of covering arrays generated by our method and the conventional method.

Those comparisons are conducted for artificial models designed based on our experience and well-known models distributed as Real-world benchmark (Cai, 2020). This benchmark contains six real-world instances. They are extracted from real test suites for Apache, Bugzilla, GCC, etc. Those models have from 20 up to 1,000 degrees with constraints.

Our approach allows us to reuse test cases defined as input covering arrays, but the reusability of test oracles along with the input covering arrays is an independent question. In order to answer RQ3, we will extend our previous work (Ukai et al., 2019) by examining various scenarios where test oracles may be reusable or not.

Since constraint handling in CIT is an area actively being studied, there are a number of techniques each of which has its own pros-and-cons in performance, flexibility, and other aspects. Hence, it is beneficial to apply “divide-and-conquer” approach to generation of a covering array so that we can utilize multiple covering array generators in combination. We will answer RQ4 by examining the detail of the procedure to employ the technique to implement the approach.

Independent variables

As mentioned already, we measure the generation time and size of output covering arrays (the dependent variables of our evaluation), for various set of settings along with different number of parameters. One suite of settings is characterized by G eneration Scenario and D esired Covering Array Model, which usually consists of D egree, R ank, S trength, and C onstraint Set. We describe each of there independent variables in our evaluation in the next sections.

Covering array generation time

Scratch Generation

Figures 6, 7 and 8 show the results of comparing the generation time between the covering arrays generated by our method and ACTS, given the strength set to 2 and the degree set up to 1,000, as it represents a large scale industrial system specification (Strength of the output covering array).

As shown in the figures, as the degree increases, our approach reduces the generation more remarkably. When the strength is 2 and degree is 980, the time is reduced by 21% to 25% or more, with or without constraint sets.

Figures 9, 10 and 11 show the results of comparing the generation time between the covering arrays generated by our method and ACTS, given the strength set to 3 and the degree set up to 380. With strength 3, we see greater time efficiency improvements than with strength 2, because the covering array generation time grows more rapidly along with the increase of degree, when a higher strength is specified. Specifically, our approach reduces the generation time by 89% to 91% compared to ACTS, when the strength is 3 and degree is 380. Also, the generation time grows more rapidly when a more complex constraint set is specified. Thus, in the scratch generation scenario, we observe the greatest improvement when a strength is set to 3 (t = 3) and the basic+ constraint set is present among the settings.

Variable strength covering array generation scenario

Figures 12, 13 and 14 show the results of comparing the VSCA (t = 2, 3) generation time between our method and ACTS, given a degree ranging from 20 to 380. Our approach reduces the generation time by 28%–30% compared to ACTS, when the mixed strengths are 2 and 3 and the degree is 380.

Figures 15, 16 and 17 show the results of comparing the VSCA (t = 2, 4) generation time between our method and ACTS, given a degree ranging from 20 to 160. Our approach reduces the generation time by up to 34% compared to ACTS, when the mixed strengths are 2 and 4 and the degree is 80.

Similar to the scratch generation scenario given a single strength, the generation time improvements are more remarkable when a higher strength is specified and a more complex constraint set is given. However, the benefit is less significant for variable strength generation unless a complex constraint (basic+) set is given.

Incremental generation scenario

Figures 18, 19 and 20 show the results of comparing the generation time between the covering arrays generated by our method and ACTS, given a degree set to 380 and the strength is 2. Our approach reduces the generation time by 84% to 98% compared to ACTS.

Similar to the scratch generation scenario, the greater generation time improvements are observed in the higher strength and with the more complex constraint sets. The improvements are more drastic than in the scratch generation scenario. This is because the conventional approach does not utilize the knowledge about the seed array which is already a covering array.

Figures 21, 22 and 23 show the results of comparing the generation time between the covering arrays generated by our method and ACTS, given a degree set to 380 and the strength is 3. In this case, our approach reduces the generation time by 99% compared to ACTS.

Generated covering array size

Reusability of test oracles by our method

Our previous work (Ukai et al., 2019) discussed how combinatorial join technique is employed to reuse test oracles over multiple software testing phases, in order to reduce total testing cost. The approach reuses the test oracles that are manually designed in the component level testing phase in later testing phases such as integration test, etc., by applying combinatorial join. The results of the previous work show that combinatorial join can reduce overall testing cost by more than 55%, and the level of reduction depends on the complexity of the software under test (SUT) and the ratio of oracle designing cost to test execution cost. However, there are several implicit assumptions behind that work, which we intend to study and discuss more in this paper, as follows:

1. Test oracles designed for one testing phase can be reused in the next testing phase.

   1. Under what conditions the SUT should display the same behavior for the reused test input?

   2. Under what conditions and what sort of bugs can be detected by reusing the test oracles in the SUT?

2. In a testing phase where these oracles are reused, none or a small amount of additional test oracles are required to be added.

We first examine these assumptions and further clarify the conditions where combinatorial join can reduce overall testing cost. For simplicity, in this discussion, we model the testing effort into two phases, “component level testing” and “system level testing”. We then evaluate our method proposed in this paper based on those conditions.

The assumption is based on a couple of other underlying assumptions: first, with the same test oracles, new bugs can be detected in later phases when a component is integrated into the original system; second, the component for which the test oracles are designed should behave in the same way as it behaves before the integration.

In general, each component is designed as much independent of each other as possible (i.e., low coupling), that is, with the minimal or none interaction between components, the integration of several component will not change the behavior of each single component. Therefore, as long as factors included in a test suite created for a certain component cover all the inputs that may affect the behaviour of that component, the oracles defined at the component level are also valid in the system level testing (1a). If a bug is detected in system level testing but not component level testing given the same test case (the same input values and the same test oracle), it means that some value combinations across multiple components are exercised in the system level testing, which is impossible to be detected inside a single component. In order to address the 1b, we can think of a few bug classes that would be detected by this approach in the system level testing, such as “resource conflict”, “incorrect abstraction”, and “unintended dependency”. Next we explain each class with examples.

Specifically, “resource conflict” refers to a type of bug that is triggered by conflicting usage of resources shared among multiple components. A list of typical bug examples of this class is shown as follows.

• Data Corruption: A component modifies shared data (such as system configuration) in a way that other components do not expect, or a component removes a directory in which other components expect their data files to be placed, etc.

• Out Of Resources: A component consumes or occupies resources (e.g., memory, disk space, network band width) more than it is allowed.

• Dead Lock: A component locks a resource (database table, file, shared memory, etc.), which other components try to access, but do not unlock it.

Oracles to detect the “Out Of Resources” and “Dead Lock” bugs are defined in a way agnostic to their input parameters. That is, for instance, an oracle for “Out Of Resource” may be described as “An out of memory error should not be thrown during a test execution”, which does not require any re-design for new input parameters and will not introduce any additional cost anyway. Therefore, among the aforementioned examples of the “resource conflict” bug class, “Data Corruption” is the only type of bugs, which can be detected by reusing test oracles through our combinatorial join approach and in which case, our approach leads to a testing cost reduction, compared to the conventional method. A bug reported by Yoonsik Park (2018) is one instance in this class, where a bug that survived all unit tests for the Linux Kernel eventually caused data corruption in the QEMU virtual machine on the kernel.

The second class of bugs under consideration is “Incorrect Abstraction”. A system sometimes has a component responsible for “abstracting” a lower level of components. For instance, a graphic card driver is such an abstracting component and a graphics card is an example of a lower level component. When another component is accessing system’s graphical capability, it expects that the capability works transparently regardless of the type of graphics card and the settings of its performance parameters. However, when an application component that utilizes the graphics capability assumes a specific behavior for the abstracting component (graphics driver), but this specific behavior is only satisfied by specific implementations (a graphics card), this class of bugs may be observed. If the specification of the abstraction component is not sufficiently defined or the testing coverage over the abstraction component is insufficient, these bugs remain undetected until the system level testing. A real-world bug found for Ubuntu (Linux) and Nvidia graphics card combination that produced unintended noise belongs to this class (Nvidia Corporation Corporation, 2019) and it could be avoided if they had an appropriate test oracle for the input.

The third class of bugs were referred as “unintended dependency”. Sometimes a component may unintentionally depend on an assumption that is sometimes broken when it is used as a part of the entire system. For instance, if a developer misses a system requirement that the product needs to be run not only on Linux but also on Windows and a file separator can only be “/”, the product will break at the system level testing, if the “OS” component is integrated in the testing phase (because path names used in the product cannot be resolved correctly). Some real-word bugs are introduced by lack of such dependency considerations (Netty Project Community, 2016; Kawaguchi, 2020). These bugs could be detected if there were test oracles for normal functionality of the SUT (i.e., checking if the Netty or Jenkins starts up and it responds to basic requests) and the test cases with these oracles were executed with a properly set-up configuration (i.e., installation= upgrade, OS= MicrosoftWindows, dotNetVersion=4.0However, the parameters came from different components ( installation mode is a parameter of Jenkins and the OS and dotNetVersion are platform parameters) and only a specific combination can trigger the bug. This means just reusing oracles is not sufficient to detect them, but guaranteeing to cover combinations between parameters is also necessary. Our method enables both without resorting to Cartesian product between two covering arrays.

The assumption 2 is satisfied if there exists a component which faces a consumer of the entire system among the components under test, and a test suite for verifying the component can also be used as a test suite for verifying the entire system. This assumption holds when the system level testing only focuses on functionality, but this is not true in general. Instead, in practice, aspects that are not examined in earlier phases, need to be more focused in later or the last testing phase (i.e., system testing), such as performance, availability, scalability, etc. Nevertheless, when a consumer facing component is present, our approach will at least reduce the cost of system level testing for the functionality aspects of the system.

In summary, reusing test oracles by our combinatorial join approach makes it possible to detect some classes of bugs in system level testing, which were not found in component level testing, without re-defining test oracles. These classes are “Data Corruption caused by Resource conflict”, “Incorrect abstraction”, and “Unintended dependencies between components”. At the same time, by reusing test oracles, functionality testing cost can be reduced in system level testing.

RQ3: What benefits does reusing test oracles across testing phases by weaken-product based combinatorial join deliver and in what conditions?

Reusing test oracles by combinatorial join can detect new bugs in system-level testing that are not found in earlier testing phases without extra manual effort. At the same time, by reusing test oracles, functionality testing cost can be reduced in system level testing.

Flexibility of weaken product combinatorial join

The combinatorial join operation produces a new covering array from two existing covering arrays, it does not create a new combination of values or handle constraints by itself. In other words, it does not matter how the existing arrays are generated. In our experiments so far, we only chose ACTS for generating the input arrays due to its popularity and high performance, but in actual use cases, any combinations of CIT tools can be utilized for the generation, depending on the actual requirements, characteristics and availability of tools, among other factors. Known CIT tools have different characteristics in performance (i.e., generation time), size efficiencies and capabilities, as described in Table 7. As we can see from the table, each tool has its own strengths and weaknesses. We summarize them as follows.

• ACTS has the best efficiency in time and size.

• PICT provides more readable notation for defining data and constraints than ACTS.

• JCUnit has the highest capability in handling various data types and constraints and its notations are the most readable among the three tools. Some of its capabilities (e.g., defining a constraint using a regular expression) cannot be replaced with neither ACTS nor PICT.

Given these characteristics, an optimal approach to build a covering array C from a complex test model (or specification) is proposed as follows, by generating sub-covering arrays first:

1. Generate a covering array A using ACTS for factors in the specification that are with constraints and can be implemented easily and directly in ACTS, or for factors that are without any constraint.

2. Generate a covering array B using JCUnit for factors in the specification that are with complex constraints that cannot be implemented directly in ACTS.

3. Combine covering arrays A and B using the combinatorial join operation.

This approach enhances the applicability of CIT in use cases where any single tool cannot generate an appropriate test suite easily, efficiently, or even possibly. For instance, if an SUT has specification that involves too complex constraints for ACTS and/or too many factors for JCUnit to generate a single covering array (test suite) all at once, this proposed approach makes it possible to use the CIT methodology for testing such SUT.

In summary, the combinatorial join operation is agnostic to how input arrays are generated and therefore it makes possible to combine multiple methods/tools to build one covering array. As shown in this discussion, there are various tools each of which has its distinct pros and cons and it is beneficial to employ the combinatorial join technique to combine covering arrays built by different tools.

RQ4: How can our approach handle constraints with flexibility?

The combinatorial join approach enables to build a covering array for a test model with numerous parameters and complex constraints using multiple CIT tools in combination, by taking full advantage of the strength of each tool, such as the high performance of ACTS and the rich constraint handling support of JCUnit.

Performance in various scenarios

In addition to the study with common settings, in this section, we study the proposed method’s performance in time and size with different settings to verify its applicability. Specifically, we study the performance of our proposed method with higher strengths, using a different covering array generation tool, and applying on real world benchmarks.

PICT

As discussed in ‘Flexibility of weaken product combinatorial join’, the proposed method can employ any CIT tool for covering array generation. To make sure our method can be applied to other tools other than ACTS, we use PICT as the underlying covering array generation engine in this experiment and measures the performance of our approach with this different generation engine. In strength 2, PICT was not able to generate covering arrays when the defined constraint sets (i.e., basic and basic+) were present even when the degree = 20 within 30 min. Also, when t = 3, it took more than 30 min to generate a covering array for degrees greater than 100. Therefore, our experiment with PICT only examine models without any constraint and we limit the degree for strength 3 up to 100.

Figures 24 and 25 show the results of comparing the generation time between our method with PICT and standalone PICT in strength 2 and 3 with no constraint) respectively. Tables 10 and 11 show the size of the generated covering arrays. We can see that the proposed method accelerates the covering array generation up to 76% and the size penalty was 16%–56% in strength 2. In strength 3, the acceleration was 96% and the size penalty was 71%. Given that the test cases are fully automated and they are executed overnight, so that the size penalty will not decrease the benefits of our approach in time reduction

Table 10:

Size of covering arrays; scratch; t = 2; d = [20, 980]; PICT.

Constraint set	None		Basic		Basic+
	min	max	min	max	min	max
PROPOSED METHOD based on PICT	61	94	N/A	N/A	N/A	N/A
PICT	39	81	N/A	N/A	N/A	N/A
Size Penalty with PICT	56%	16%	N/A	N/A	N/A	N/A

DOI: 10.7717/peerjcs.720/table-10

Table 11:

Size of covering arrays; scratch; t = 3; d = [20, 100]; PICT.

Constraint set	None		Basic		Basic+
	min	max	min	max	min	max
PROPOSED METHOD based on PICT	363	363	N/A	N/A	N/A	N/A
PICT	226	692	N/A	N/A	N/A	N/A
Size Penalty with PICT	60%	71%	N/A	N/A	N/A	N/A

DOI: 10.7717/peerjcs.720/table-11

Summary and discussion

First of all, our proposed method offers a way to reuse test oracles designed in earlier testing phases (e.g., unit testing, component testing) in later ones such as integration and system testing. Moreover, the method can accelerate covering array generation under complex constraint sets. It also enhances applicability of combinatorial interaction testing tools with different strengths and weaknesses in flexible combination, since the method is transparent to underlying generation algorithms.

Second, our proposed method accelerates an existing covering array generation algorithm by combining outputs of it, at the cost of increase in output size (i.e., size penalty). In general, The size penalty becomes smaller and the time reduction becomes greater when the constraint set is more complex and the degree is larger. Specifically, the generation time reduction varies from 13% to 99% depending on generation scenarios and degrees of the method’s output. The increase in size can go up to 141% in the worst case. Although the increase in size is big in some cases and it seems a concern to apply the approach, it is still beneficial, from different aspects in different situations, as discussed as follows.

A first simple practical use case is when the test execution time matters less than the test generation time, our proposed method will be useful, from a comprehensive perspective. This is common in modern software development projects, in which test execution is highly automated not only in unit testing but also in later testing phases such as integration testing and system testing.

Also, as shown in Figs. 6–11, the generation time of the conventional tool (i.e., ACTS) grows more rapidly than linear, along with the degree increases, it becomes impractical quickly as the degree increases. Our approach first uses the tool to generate two smaller covering arrays and combines them later. This approach enhances the applicability of current generation tools (such as ACTS) to scenarios where it has not been practical due to too many parameters (i.e., very high degree) and too long generation time. But with our approach, the large number of parameters are split into two sets, and the generation tool only handles half of the parameters, which may largely reduce the generation time and make it practical.

There are situations where the cost to change a value for a testing parameter varies largely. For example, some parameters require OS re-installation while others can be changed by just operating an application. If a single covering array (test suite) is generated for all the parameters at once, each test case may have different values of OS parameters, that may require an OS re-installation before executing each single test case, which can be very expensive. But our proposed approach can generate covering arrays for OS parameters and other application parameters separately and combine them into one later. This will largely reduce the switching of OS parameters (i.e., OS re-stalling) to a minimal numbers. Since our method does not create a new row, the overall test execution cost will be reduced even with some size penalty, because the execution time of additional test cases is still much cheaper than the cost of many more times of OS re-installation.

When a user needs to add some parameters to an existing test suite, “seeding” functionality of a CIT tool has been used. As shown in Figs. 19 and 20, the generation time is dramatically reduced when our method is applied to this use case. This is because the conventional method (ACTS) needs to examine coverage of the input covering array first, which is time consuming and unnecessary for our method. Since the size penalty for this use case is relatively modest (30%–60%), if test case design time matters more than execution time because of testing automation, for instance, it will be a practical solution.

Besides, more generally, in a situation where the constraints are more complex or the degree is larger, our proposed method shows more significant benefit, because the size penalty decreases and the time reduction increases as the constraint set becomes more complex and the degree increases.

The aforementioned situations show that our proposed method is beneficial, even with large size penalty at times.

Threats to validity

In this section, we discuss some threats to validity in our study.

Future work

Our approach assumes that there is no constraint defined across LHS and RHS. However, it is usual not to have such an assumption in practical situations, especially when we construct a VSCA for a system with multiple components. From the technical point, sometimes it is even impossible to define a combinatorial join operation when constraints across LHS and RHS are present. For instance, if the strength of LHS and RHS is t and there is a constraint across them which involves more than t parameters in either LHS or RHS, there might not exist sufficient rows to cover all t-way tuples or even any row that satisfies the constraint at all. As one of our future works, we intend to study the exact criteria where the operation can be meaningful, and design an efficient algorithm to perform the operation under the situation that satisfies such criteria.

Our approach generates covering arrays of larger size than other tools, particularly when the strength is higher than 2. As one of the future work, we intend to apply a squashing technique to diminish a redundant covering array.

Lastly, our current algorithm is sufficiently fast in strength 2 and 3, but it may become less efficient in strength 4 or greater. It is known that a bug can be found in a strength up to 6 or 7 (Kuhn, Kacker & Lei, 2016). Therefore, in order to improve the applicability of our approach in practice in high strength, our algorithm needs further improvement.