Multi-objective test selection of smart contract and blockchain applications

Background

In this section, we provide a brief overview of the main concepts in block chain technology and SC.

 
 
_______________________1pragma  solidity  ˆ0.6.0; 
_________________________________________________________________________________ 2 
__________________________________________________________________________________________________________________________________________________ 3contract  MembershipSubscription  { 
_________________________________________________________________________________ 4 uint256  monthlyFees ; 
_________________________________________________________________________________ 5 
__________________________________________________________________________________________________________________________________________________ 6 constructor ( uint256  fee ) public { 
_________________________________________________________________________________ 7 monthlyFees  = fee ; 
_________________________________________________________________________________ 8 } 
_________________________________________________________________________________ 9 
__________________________________________________________________________________________________________________________________________________ 10 function  makePayment () payable  public  { 
_________________________________________________________________________________ 11 } 
_________________________________________________________________________________ 12 
__________________________________________________________________________________________________________________________________________________ 13 function  isBalanceValid ( uint256  monthsElapsed ) public  view 
_________________________________________________________________________________  returns  ( bool ) { 
_________________________________________________________________________________ 14 return  monthlyFees  * monthsElapsed  >= address ( this ) . balance 
_________________________________________________________________________________  ; 
_________________________________________________________________________________ 15 } 
_________________________________________________________________________________ 16 
__________________________________________________________________________________________________________________________________________________ 17 function  withdrawBalance () public  { 
_________________________________________________________________________________ 18 msg . sender . transfer ( address ( this ) . balance ) ; 
_________________________________________________________________________________ 19 } 
_________________________________________________________________________________ 20} 
                       Listing 1: Sample Smart contract    

Related work

Approach overview

The main objective of our approach is to optimize a test suite to achieve three goals as illustrated in Fig. 1. Initially, we take the SC in addition to its existing test cases as inputs. Our algorithm will begin by analyzing and collecting certain data such as coverage, execution time and cost, test case size, and so on. Our objective is to maximize coverage, reduce total execution time, and minimize the gas cost. Since these objectives are conflicting, and because we are inherently dealing with a huge search space, we will utilize a multi-objective algorithm to locate the Pareto-optimal solutions for this problem. An overview of this algorithm and how it can be used in our case will be illustrated in the subsequent section.

Search-based formulation

 
_______________________________________________________________________________________________________ 
Algorithm 1 Pseudo code of NSGA-II adaptation for smart contract test-cases 
prioritization______________________________________________________________________________________ 
 1:  Inputs: Solidity smart contract P, Test suite TC 
 2:  Output: subset(s) of the test suite 
 3:  Begin 
 4:  I:= Instantiation(TC)// vectors of TCs 
 5:  P0:= set_of(I) 
 6:  t:= 0 
 7:  Repeat 
 8:  Ct:= apply_Genetic_Operators(Pt) 
 9:  Gt:= Pt ∪ Ct // Combine parent and offspring populations 
10:  for all I ∈ Gt do 
11:     Execution_Time(I):= calculate_Execution_Time(P) 
12:     Coverage(I):= calculate_Code_Coverage(P) 
13:     Cost(I):= calculate_Execution_Cost(P) 
14:  end for 
15:  F:=    fast_Non_Dominated_Sort(Gt)    // F=(F1,F2, ...), all nondominated 
   fronts of Gt 
16:  Pt+1 = ∅ 
17:  i:= 1 
18:  while  |Pt+1| + |Fi| < Max_size do 
19:     Crowding_distance_assignment(Fi) // calculate crowding distance in Fi 
20:     Pt+1= Pt+1 ∪ Fi // include ith nondominated front in parent pop 
21:     i:= i+1 
22:  end while 
23:  Sort (Fi, ≺n) // sort in descending order using ≺n 
24:  Pt+1=  Pt+1  ∪ Fi  [1. . . (Max_size −|Pt+1|)]  // choose the first Max_size - 
   |Pt+1| elements of Fi 
25:  t:= t+1 // increment generation counter 
26:  until t=Max_iteration 
27:  best_solutions := first_front(Pt) 
28:  return best_solutions______________________________________________________________________________    

In the literature (Mkaouer et al., 2015; Alkhazi et al., 2020b), the multi-objective problem as follows is represented by the following formula: ${\displaystyle \left\{\begin{array}{c}Minf\left(x\right)={\left[f_1\left(x\right),f_1\left(x\right),\dots ,f_M\left(x\right)\right]}^T\hfill \\ g_j\left(x\right)\ge 0\hfill & j=1,\dots ,P;\hfill \\ h_k\left(x\right)=0\hfill & k=1,\dots ,Q;\hfill \\ x_i^L\le x_i\le x_i^U\hfill & i=1,\dots ,n;\hfill \end{array}\right.}$

In short, the number of objective functions is represented by M. Whereas Q and P are the equality constraints and inequality constraints, respectively. The lower bound of the decision variable is represented by $x_i^L$, and the higher ones by $x_i^U$.

Each solution x is represented by a number of decision variables. The metaheuristic algorithm objective is to optimize these variables. Ω is a set of all feasible solutions (search space), which are the solutions that met the constraints ((P + Q)). To calculate the objective value for a particular solution f_i, the fitness function f is evaluated where all objectives should be minimized. When the maximum value of an objective is desired, we simply take its negative value to meet the algorithm’s condition. A high-level pseudo-code of NSGA-II is shown in Algorithm 1.

Pareto-optimal solutions

For each multi-objective problem, we evaluate its defined objective functions for a specific solution. By comparing the objective vectors of two solutions, we can figure out which one is ‘better’ based on these objectives. One common way to do this comparison is by adding up all the objective values of one solution and comparing the total with that of another solution. However, this only works if all the values are in the same units of measurement. In the context of SBSE, we frequently rely on the concept of Pareto optimality. As defined in (1) and (2) with strict inequality (Harman, 2007), Pareto optimality means that a solution is considered superior to another if it outperforms it in at least one objective function and doesn’t perform worse in any other. This definition helps us identify which solution is better, although it doesn’t provide a measure of ‘how much’ better it is. (1)${\displaystyle F\left(\overline{\mathrm{x1}}\right)⩾F\left(\overline{\mathrm{x2}}\right)⟺{\forall }_i{f}_i\left(\overline{\mathrm{x1}}\right)⩾f_i\left(\overline{\mathrm{x2}}\right)}$ (2)${\displaystyle F\left(\overline{\mathrm{x1}}\right)>F\left(\overline{\mathrm{x2}}\right)⟺{\forall }_i{f}_i\left(\overline{\mathrm{x1}}\right)⩾f_i\left(\overline{\mathrm{x2}}\right)\wedge {\exists }_i{f}_i\left(\overline{\mathrm{x1}}\right)>f_i\left(\overline{\mathrm{x2}}\right)}$

In the field of SBSE, the algorithms utilize the concept of Pareto optimality as part of the search process to generate a collection of non-dominated solutions. Each of these non-dominated solutions can be thought of as a balanced trade-off across all objective functions, where no solution in the set is definitively better or worse than another. It’s important to acknowledge that SBSE operates under the assumption that accurately determining the ‘true’ Pareto front of a problem—which is the set of all values that are Pareto optimal—is analytically infeasible and impractical to achieve through exhaustive search. Consequently, every set produced through metaheuristic search serves as an approximation of this often elusive ‘true’ Pareto front (Fig. 2). Further iterations of such an algorithm could potentially enhance this approximation.

Solution representations

Table 1 is a sample solution vector. Each solution is represented by a vector where every dimension constitutes a test case. After analyzing each test case to measure its execution time, gas cost, and coverage, we use this data in the solution representation as shown in Table 1 where each test case constitutes: Test_Case(ID, Gas Cost, Execution Time, and Covered Statements.)

 
 
________________________________________________________________________________________________________________ 1//  SPDX - License - Identifier :  GPL -3.0 
_________________________________________________________________________________ 2pragma  solidity  ˆ0.8.4; 
_________________________________________________________________________________ 3 
__________________________________________________________________________________________________________________________________________________ 4contract  donations { 
_________________________________________________________________________________ 5 struct  Donation  { 
_________________________________________________________________________________ 6 uint  id ; 
_________________________________________________________________________________ 7 uint  amount ; 
_________________________________________________________________________________ 8 string  donor ; 
_________________________________________________________________________________ 9 uint  timestamp ; // seconds  since  unix  start 
_________________________________________________________________________________ 10 } 
_________________________________________________________________________________ 11 uint  amount  = 0; 
_________________________________________________________________________________ 12 uint  id  = 0; 
_________________________________________________________________________________ 13 mapping ( address  => uint ) public  balances ; 
_________________________________________________________________________________ 14 mapping ( address  => Donation []) public  donationsMap ; 
_________________________________________________________________________________ 15 
__________________________________________________________________________________________________________________________________________________ 16 function  donate ( address  _recipient , string  memory  _donor ) 
_________________________________________________________________________________  public  payable  { 
_________________________________________________________________________________ 17 require ( msg . value  > 0, " The  donation  needs  to  be  >0  in 
_________________________________________________________________________________  order  for  it  to  go  through " ) ; 
_________________________________________________________________________________ 18 amount  = msg . value ; 
_________________________________________________________________________________ 19 balances [ _recipient ] += amount ; 
_________________________________________________________________________________ 20 donationsMap [ _recipient ]. push ( Donation ( id ++, amount , _donor , 
_________________________________________________________________________________  block . timestamp ) ) ; 
_________________________________________________________________________________ 21 } 
_________________________________________________________________________________ 22 
__________________________________________________________________________________________________________________________________________________ 23 function  withdraw () public  { // whole  thing  by  default . 
_________________________________________________________________________________ 24 amount  = balances [ msg . sender ]; 
_________________________________________________________________________________ 25 balances [ msg . sender ] -= amount ; 
_________________________________________________________________________________ 26 require ( amount  > 0, " Your  current  balance  is  0 " ) ; 
_________________________________________________________________________________ 27 ( bool  success ,) = msg . sender . call { value : amount }( " " ) ; 
_________________________________________________________________________________ 28 if (! success ) { 
_________________________________________________________________________________ 29 revert () ; 
_________________________________________________________________________________ 30 } 
_________________________________________________________________________________ 31 } 
_________________________________________________________________________________ 32 
__________________________________________________________________________________________________________________________________________________ 33 function  balances_getter ( address  _recipient ) public  view 
_________________________________________________________________________________  returns  ( uint ) { 
_________________________________________________________________________________ 34 return  balances [ _recipient ]; 
_________________________________________________________________________________ 35 } 
_________________________________________________________________________________ 36 
__________________________________________________________________________________________________________________________________________________ 37 function  getBalance () public  view  returns ( uint ) { 
_________________________________________________________________________________ 38 return  msg . sender . balance ;  
_________________________________________________________________________________ 39 } 
_________________________________________________________________________________ 40} 
Listing 2:  Sample smart contract adopted from Remix documentation Remix 
(2021)    

 
 
_______________________________________________________________________________________________________ 1//  SPDX - License - Identifier :  GPL -3.0 
_________________________________________________________________________________ 2pragma  solidity  ˆ0.8.4; 
_________________________________________________________________________________ 3import  " ./ donations . sol " ; 
_________________________________________________________________________________ 4 
__________________________________________________________________________________________________________________________________________________ 5contract  testSuite  is  donations  { 
_________________________________________________________________________________ 6 address  sender  = TestsAccounts . getAccount (0) ; 
_________________________________________________________________________________ 7 address  recipient  = TestsAccounts . getAccount (1) ; 
_________________________________________________________________________________ 8 
__________________________________________________________________________________________________________________________________________________ 9 function  donateAndCheckBalance () public  payable { 
_________________________________________________________________________________ 10 Assert . equal ( msg . value , 1000000000000000000 , ’ value  should 
_________________________________________________________________________________  be  1  Eth ’ ) ; 
_________________________________________________________________________________ 11 donate ( recipient , " Bader " ) ; 
_________________________________________________________________________________ 12 Assert . equal ( balances_getter ( recipient ) , 
_________________________________________________________________________________  1000000000000000000 , ’ balances  should  be  1  Eth ’ ) ; 
_________________________________________________________________________________ 13 } 
_________________________________________________________________________________ 14 
__________________________________________________________________________________________________________________________________________________ 15} 
            Listing 3: Sample test case for the contract in listing 2    

A simple Solidity contract is shown in listing 2, and a sample test case for this contract is shown in listing 3. Executing this test case will cover 50% of the statements leaving half of the functions untested. While executing this single test case required only 0.86 s on a local test BC, executing a transaction on the mainnet requires 15 s on average. The speed of the transaction on the mainnet is affected by different factors such as network congestion, paid gas, and number of available miners. The consumed gas for this single test case was 21,484. Therefore, executing this test case on the mainnet will cost around $4.0 at the current average gas price (4∼6 Gwei), making running more cases a burden on the testing team.

In general, the initial population is selected randomly, and the vector V is bound by a maximum size V_MAX that is proportional to the SC’s statements and the test cases available for selection.

Solution variation

Variation operations are vital in exploring the search space. They allow the algorithm to potentially find better alternative solutions by using the crossover, mutation, and selection of the fittest. In our case, we will use a one-point crossover operation where the parent solutions are split at an arbitrary location. The next step is crossing the split parts between the parents to form new children as shown in Fig. 3. The crossover allows for better convergence in a subspace. The mutation, however, helps to diversify the population and provides a way to escape a local optimum. In our study, we use the bit-string mutation operator to randomly select one test case in the vector dimension in order to replace it with another entry. Figure 4 illustrates a high-level overview of the mutation operator.

Evaluation metrics

During this step, we make sure that we select the fittest solutions (elitism) to be carried out for the next iteration. Therefore, we use three fitness functions to evaluate and rank solutions to make a better decision about whether a solution should pass to the next generation or not.

Lower gas cost: To perform an operation on the Ethereum BC (or almost any BC), a particular amount of gas needs to be paid. The motivation for this fee is twofold. First, the BC will be free from bad actors and spammers who may flood the BC to cause congestion and breakdowns. This is because the transaction originator should specify the amount of Ether they are willing to pay during the life cycle of their transactions. Therefore, bad actors need a significant amount of capital to flood the network. The same applies to poor programming practices such as infinite loops, where the transaction would eventually fail and throw an out-of-gas exception. The second reason for the gas fees is to pay the miners for processing the transactions and including it in the block. The transaction cost is determined by two factors: amount of gas used, and the gas price as shown in the following formula: (3)${\displaystyle GasCost=Used\text{\_}Gas\ast Gas\text{\_}Price}$

The goal is to organize the test cases in a way where testers who are on budget can run a larger number of test cases. It is worth mentioning that gas price varies over time based on the number of available miners, network congestion, and Ether price.

Maximum coverage: The objective is simply to increase the SC coverage. After executing each test case, we trace the triggered code statements by that case. We later combine the aggregated covered statements to measure the coverage score. It is important to note that coverage and gas costs are not correlated. In solidity, some statements and functions will cost lower gas than others since they do not access the BC state. For instance, Pure functions in Solidity are purely computational and deterministic. They only operate on input parameters to compute a value and do not modify the state of the BC. As a result, they are less expensive in terms of gas than functions that do modify the state. Thus, covering these functions in test cases can improve code coverage without significantly increasing gas consumption.

Lower execution time: A shorter execution time is desired; therefore, we aim to minimize the time needed to discover faults. We will use a test net to run every test case and record its execution time. This data will be used in the following step to prioritize test solutions with lower execution time.

Research Questions

To evaluate our approach, we defined three research questions as follows:

• RQ1: Search validation (sanity check). To assess the need of our problem formulation, we compared our multi-objective approach to a random search algorithm (RS). If RS proves to be more effective than a sophisticated search method, it would indicate that there is no need for the use of metaheuristic search.

• RQ2: How efficient is the proposed multi-objective method in comparison to mono-objective approach? To confirm that the objectives are in opposition, we combined the three normalized objectives into a single fitness function. If the results are equal or the mono-objective approach outperforms the multi-objective approach, it suggests that the latter is unnecessary. The single-objective genetic algorithm, which has the same design as NSGA-II but only for optimizing a single objective, was used for the mono-objective approach. It does not utilize the non-dominance principles and crowding distance in generating the Pareto front.

• RQ3: How effective is our multi-objective test case selection method in revealing faults? A well-designed test selection approach plays a vital role in ensuring the success of a software system by striking a balance between cost and time efficiency and thorough fault detection. By carefully selecting the tests to be executed, this approach minimizes the cost and time needed to run the tests, while preserving the test suite’s effectiveness in uncovering faults, thereby maintaining the validity of the software’s functionality and performance.

Case Studies

To evaluate the research questions, we used five solidity SC case studies based on data collected from GitHub and State of the DApps (https://www.stateofthedapps.com). We only considered projects that are hosted in the top five BC platforms, written in Solidity, and has its open-source code for the SC published. We did not include stealth, inactive, or beta projects. In addition, projects with less than 100 lines of code for their combined SC was not considered for evaluation.

The case studies employed vary in size, application area, and structure. The solidity programs that we used are briefly described in the section below.

• Chainlink is a decentralized oracle network that connects SC to real-world data, events, and payments. It enables SC to securely access off-chain data feeds, web APIs, and traditional bank payments. This allows for the creation of more advanced and reliable SC, which can be used for a wide range of purposes such as financial applications, supply chain management, and gaming. The Chainlink network is composed of a network of independent, security-reviewed node operators that provide data to SC, and is secured by a decentralized network of nodes. The token of Chainlink (LINK) is used as a form of payment to the node operators for providing these services. The project is open-source and is supported by a large and active community of developers and users.

• PancakeSwap is a decentralized exchange (DEX) built on the Binance Smart Chain (BSC) that allows users to trade various tokens built on the Binance ecosystem. The exchange utilizes an automated market maker (AMM) model, where users can trade tokens without the need for a traditional order book. Additionally, PancakeSwap features a unique liquidity mining mechanism, where users can provide liquidity to the exchange and earn rewards in the form of CAKE tokens. This incentivizes users to provide liquidity to the exchange, increasing its overall liquidity and contributing to its overall decentralization. The platform is fully decentralized and is governed by its community of users through a decentralized autonomous organization (DAO). PancakeSwap has quickly gained popularity among the decentralized finance (DeFi) community for its user-friendly interface, low transaction fees, and high liquidity.

• Venus Protocol The Venus Protocol is a decentralized finance (DeFi) platform built on the Binance Smart Chain. It aims to provide a stablecoin-based yield farming and lending platform, where users can lend and borrow assets, earn interest, and participate in liquidity provision. The Venus Protocol uses a collateralized stablecoin, called VAI, as its base currency, which is pegged to the value of the US dollar. This allows for more stable lending and borrowing rates, as well as reducing the volatility of returns for yield farming. Additionally, the Venus Protocol features a unique governance system, where users can vote on and propose changes to the protocol using their XVS tokens. This allows for a more decentralized and community-driven approach to the development and management of the protocol. The Venus Protocol aims to provide a fair, transparent, and sustainable DeFi platform for users to earn returns on their assets and participate in the growth of the ecosystem.

• Axie Infinity is a play-to-earn decentralized gaming platform built on the Ethereum BC. It allows players to earn cryptocurrency by participating in the game, such as by breeding valuable Axies or participating in in-game events. The game features unique collectible creatures called Axies that can be bought, bred, and battled. The genetic system allows players to breed different Axies and create new and unique offspring. It also has a marketplace where players can buy, sell and trade different Axies. The platform also features a unique governance system where players can vote on and propose changes to the game using their SLP tokens. The game has a vibrant community and is actively developed by the team behind it.

• SushiSwap is a decentralized exchange (DEX) built on the Ethereum BC that allows users to trade various tokens, including Ethereum and other ERC-20 tokens. The exchange utilizes an automated market maker (AMM) model, where users can trade tokens without the need for a traditional order book. It also features a unique liquidity mining mechanism, where users can provide liquidity to the exchange and earn rewards in the form of SUSHI tokens. This incentivizes users to provide liquidity to the exchange, increasing its overall liquidity and contributing to its overall decentralization. SushiSwap also supports yield farming, where users can earn additional rewards by providing liquidity to certain pools. The platform is fully decentralized and is governed by its community of users through a decentralized autonomous organization (DAO). SushiSwap gained popularity among the decentralized finance (DeFi) community for its unique features, low transaction fees, and high liquidity.

Table 2 summarizes the structural details of the test cases we used in our study. The table shows the number of SC for each project, the total number of functions and branches, and the total number of lines of code (LoC). In Table 3, however, we describe the test cases available for selection for each project, the time needed to execute all cases in a “test network”, the required amount of budget to cover the gas requirement, and the percentage of coverage achieved for both lines and methods if we execute all available test cases. We used the test suite created and used by the developers of each project. To extract the required coverage details, we used both Hardhat (https://hardhat.org), and Solidity-coverage (https://github.com/sc-forks/solidity-coverage). Moreover, to get an estimate of the required execution time and gas for each method, we used Gas-reporter (https://github.com/cgewecke/hardhat-gas-reporter) tool.

Experimental settings

The effectiveness of search algorithms can be greatly impacted by the settings of parameters, as noted by Arcuri & Fraser (2013). It is crucial to choose the appropriate population size, stopping criterion, crossover rate, and mutation rate to prevent early convergence. In our experiments, we utilized MOEA Framework v3.2 (Hadka, 2012), and conducted several trials with varying population sizes of 50, 100, 250, and 500. The stopping criterion was fixed at 100k evaluations for all algorithms. For crossover and mutation, probabilities of 0.5 were used for both, per generation. We employed the Design of Experiments (DoE) approach, which is one of the most efficient and widely used methods for parameter setting in evolutionary algorithms, as outlined by Talbi (2009). Each parameter was uniformly discretized into certain intervals, and values from each interval were tested for our application. Following multiple trial runs, we established the parameter values for the algorithm as 100 solutions per population and a maximum of 300 generations.

We used the MOEA Framework’s standard parameter values for all other parameters. Since metaheuristic algorithms are probabilistic optimizers, they may produce different outcomes for the same problem. Therefore, we conducted 30 independent runs for each configuration and problem instance. The results were then analyzed statistically using the Wilcoxon test, as suggested by Arcuri and Fraser, with a confidence level of 95% (α = 5%). All experiments were carried out on a Macbook Pro machine equipped with a 2.3 GHz Intel 8-Core i9 processor, and 16 GB 2400 MHz DDR4 RAM. The Hardhat v2.6.8, Solc version: 0.8.16, solidity-coverage v0.8.2, and hardhat-gas-reporter v1.0.9 were utilized during the experiments. The Ethereum price at the time of writing the paper was $1,588, and the gas price was set at 85 GWEI.

The primary goal of comparing the mono-objective search is to determine if the three objectives are in conflict. Therefore, we assigned equal weight to all objectives after normalization between 0 and 1, for the single-objective formulation. The mono-objective approach only generates one solution, while the multi-objective algorithm generates multiple non-dominated solutions that spread across the Pareto front of the objectives. To ensure fair and meaningful comparisons, we employed the knee-point strategy to select the NSGA-II solution for the multi-objective algorithm (Branke et al., 2004). The knee point represents the solution with the highest trade-off between the different objectives, and it can be considered equivalent to the mono-objective solution with equal weights for the objectives if there is no conflict among them. The knee point was chosen from the Pareto approximation by identifying the median hyper-volume IHV value. Assigning weights to objectives is often difficult as it depends on the developers’ priorities and preferences. A multi-objective approach eliminates the need to assign weights to objectives, as developers can select a solution by visualizing the Pareto front based on the objectives. The knee-point method was adopted to guarantee an unbiased comparison between the two algorithms, as it is considered best practice according to the present state of computational intelligence.

Evaluation metrics

To evaluate our approach and to answer RQ1, we evaluate our NSGA-II formulation against random search (RS) in terms of (1) search algorithm efficiency and (2) test optimization system performance. The aim is to confirm the necessity of an intelligent search method. This RQ serves as a basic verification and typical benchmark inquiry in any SBSE formulation effort (Harman & Jones, 2001). If the intelligent search method doesn’t surpass random search, it indicates that the proposed formulation is inadequate.

To assess the search algorithm performance, we quantify the efficiency of each algorithm in exploring the search space. Multi-objective evolutionary algorithms, unlike mono-objective ones, produce a set of non-dominated or Pareto optimal solutions accumulated during the search process. Thus, we use three standard metrics for evaluating multi-objective optimization algorithms: Hypervolume, Spread and Generational Distance (Zitzler et al., 2003).

• Hypervolume (HV): is a performance indicator used to measure the quality of a set of solutions obtained by an algorithm. It quantifies the volume of the dominated space in the objective space, where the solution set dominates the area. A larger hypervolume value indicates that the algorithm has found a set of solutions that cover a greater area in the objective space, hence, a higher quality set of solutions. HV is used to compare the performance of different algorithms and to determine the trade-off between conflicting objectives.

• Generational distance (GD): is another performance indicator used in multi-objective evolutionary algorithms. It measures the average Euclidean distance between a set of solutions generated by an algorithm and a set of reference solutions, known as the Pareto front. The reference solutions are considered to be the optimal solutions, and GD provides a way to measure how close the algorithm’s solutions are to the optimal solutions. A smaller GD value indicates that the algorithm’s solutions are closer to the optimal solutions and hence, of higher quality. GD is used to assess the convergence of the algorithm towards the Pareto front.

In regard to RQ2, we defined a mono-objective algorithm that generates only one solution as output, formed by aggregating the three normalized objectives into a single fitness function. In order to measure the performance of both algorithms in solving our problem, i.e., test case selection, we compare the results in terms of cost, time, and coverage in addition to defining two more metrics: (4)${\displaystyle Cost\text{\_}per\text{\_}FC\text{\_}point=\left[\frac{New\text{\_}Cost\left(\text{\%}\right)}{FC\left(\text{\%}\right)}\right]}$ (5)${\displaystyle Cost\text{\_}per\text{\_}time\text{\_}point=\left[\frac{New\text{\_}Cost\left(\text{\%}\right)}{Saved\text{\_}Execution\text{\_}time\left(\text{\%}\right)}\right]}$

These metrics will better illustrate the efficiency of each algorithm in maximizing the coverage while minimizing the gas cost and computational time.

RQ3: Our approach focuses on streamlining the test suite for the post-deployment phase, supplementing the extensive initial testing done before production. Thus, it is not anticipated for the reduced test suite to exhibit the same level of performance as the original suite, as the emphasis will be on examining the portions of the code that are more susceptible to failures due to the dynamic nature of the Ethereum network.

This RS investigates the effectiveness of the new test suite in detecting most of the critical faults? To do so, we will create various mutations to induce bugs at multiple locations. We will use the approach and mutation operators proposed by Wu et al. (2019a). In summary, there are traditional mutation operators (Table 4) and Solidity specific ones (Table 5). The specific mutation operators are grouped into four groups: (1) keyword operators, (2) global variables and functions operators, (3) variable unit operators, and (4) error handling operators. For instance, functions in a contract can be declared as view or pure. A “view” function only reads data and does not modify the contract’s state or emit events. “Pure” functions, on the other hand, are restricted from both reading from and modifying the state, and can only call other “pure” functions. The function state keyword change (FSC) operator can alter the behavior of a function by switching the “view” keyword to “pure,” thus changing its state. A sample function state keyword change (FSC) is shown in Table 6. Another example of a mutation operator is variable type keyword replacement (VTR). Solidity is a statically and strongly typed language, meaning that the data type of each variable must be specified. An integer overflow can result in the loss of the most significant bits of the result, creating real-world security vulnerabilities. Table 7 provides an illustration of a VTR mutant, which requires testers to take negative numbers and truncation into account during testing. Further discussion about the approach, operators, and their impact on the SC are presented in Wu et al. (2019a). After creating various mutations using both general and Solidity specific operators for each test case, we compare the performance of each algorithm in terms of its effectiveness in revealing faults using the following formula: (6)${\displaystyle Effectiveness=\left[\frac{DetectedFaults}{TotalFaults}\right]\ast 100}$

Moreover, we calculate test case fault detection rate (DR) per unit of time as follows: (7)${\displaystyle DR=\left[\frac{DetectedFaults}{Time\left(s\right)}\right]}$

These metrics will demonstrate the effectiveness and efficiency of each algorithm in choosing test cases that uncover a higher percentage of severe faults.