Faster numeric static analyses with unconstrained variable oracles

A dataflow analysis for LU variables

We now informally sketch several variants of a forward dataflow analysis for the computation of LU variables, to be used on a control flow graphics (CFG) representation of the source program; if needed, the approach can be easily adapted to work with alternative program representations.

The transfer function for non-relational analyses. Let $Stmt$ be the set of statements occurring in the CFG basic blocks, which for simplicity we assume to resemble 3-address code. Then, given the set $lu\subseteq Var$ of variables that are LU before (abstractly) executing $s\in Stmt$, the transfer function

$$\mathrm{⟦}\cdot \mathrm{⟧}:Stmt\times \mathrm{\wp }(Var)\to \mathrm{\wp }(Var)$$computes the set ⟦s⟧(lu) of variables that are LU after the execution of $s$. Clearly, the definition of $\mathrm{⟦}\cdot \mathrm{⟧}$ depends on the target analysis: a more precise abstract domain will probably expose fewer LU variables. We first consider, as the reference target analysis, the non-relational abstract domain of intervals (Cousot & Cousot, 1977). Intuitively, in this case a variable is LU if the corresponding interval is (likely) unbounded, i.e., $[-\mathrm{\infty },+\mathrm{\infty }]$.

In our definitions, we explicitly disregard those constraints that can be implicitly derived from the variable datatype; for instance, for a nondeterministic assignment $(x\leftarrow ?)\in Stmt$, even when knowing that $x\in Var$ is a signed integer variable stored in an 8-bit word, we will ignore the implicit constraints $-128\le x\le 127$ and flag the variable $x$ as LU. Hence, the transfer function for nondeterministic assignments is

(1) $$\mathrm{⟦}x\leftarrow ?\mathrm{⟧}(lu)=lu\cup \{x\}.$$

The transfer function for the assignment of a constant value $k\in \mathbb{Z}$ to a variable $x\in Var$ is simply defined as

(2) $$\mathrm{⟦}x\leftarrow k\mathrm{⟧}(lu)=lu\backslash \{x\},$$meaning that after the assignment $x$ is constrained (and hence removed from set $lu$). Similarly, when the right hand side of the assignment is a variable $y\in Var$, we can define

(3) $$\mathrm{⟦}x\leftarrow y\mathrm{⟧}(lu)=\{\begin{array}{cc}lu\cup \{x\},\hfill & \mathrm{i}\mathrm{f}y\in lu;\hfill \\ lu\backslash \{x\},\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}$$

A more interesting case is the transfer function for the assignment statement $(x\leftarrow yopz)\in Stmt$, where $op\in \{+,-,\ast ,/,\mathrm{\%}\}$ is an arithmetic operator and $x,y,z\in Var$, which is defined as follows:

(4) $$\mathrm{⟦}x\leftarrow yopz\mathrm{⟧}(lu)=\{\begin{array}{cc}lu\backslash \{x\},\hfill & \mathrm{i}\mathrm{f}y\notin lu\mathrm{a}\mathrm{n}\mathrm{d}z\notin lu,\hfill \\ \hfill & \mathrm{o}\mathrm{r}\mathrm{i}\mathrm{f}op=\mathrm{\%}\mathrm{a}\mathrm{n}\mathrm{d}\mathit{z}\notin \mathit{l}\mathit{u}\mathit{,}\hfill \\ \hfill & \mathrm{o}\mathrm{r}\mathrm{i}\mathrm{f}op=-\mathrm{a}\mathrm{n}\mathrm{d}y=z;\hfill \\ lu\cup \{x\},\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}$$

Namely, $x$ is going to be constrained when both $y$ and $z$ are constrained, or when $z$ is constrained and $op$ is the modulus operator, or when hitting the corner case $x\leftarrow y-y$. For the special case when the third variable $z$ is replaced by a constant argument $k\in \mathbb{Z}$, we can define

(5) $$\mathrm{⟦}x\leftarrow yopk\mathrm{⟧}(lu)=\{\begin{array}{cc}lu\backslash \{x\},\hfill & \mathrm{i}\mathrm{f}y\notin lu,\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{f}op=\mathrm{\%},\hfill \\ \hfill & op=\ast \mathrm{a}\mathrm{n}\mathrm{d}k=0;\hfill \\ lu\cup \{x\},\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}$$

When evaluating Boolean guards, the abstract semantics works in a similar way: letting $(x\bowtie y)\in Stmt$, where $x,y\in Var$ and $\bowtie \in \{<,\le ,=,\ge ,>\}$, we can define

(6) $$\mathrm{⟦}(x\bowtie y)\mathrm{⟧}(lu)=\{\begin{array}{cc}lu\backslash \{x\},\hfill & \mathrm{i}\mathrm{f}x\in lu\mathrm{a}\mathrm{n}\mathrm{d}y\notin lu;\hfill \\ lu\backslash \{y\},\hfill & \mathrm{i}\mathrm{f}y\in lu\mathrm{a}\mathrm{n}\mathrm{d}x\notin lu;\hfill \\ lu,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}$$

As before, when variable $y$ is replaced by a constant argument $k\in \mathbb{Z}$ we can refine our transfer function as follows:

(7) $$\mathrm{⟦}(x\bowtie k)\mathrm{⟧}(lu)=lu\backslash \{x\}.$$

Similar definitions can be easily provided for all the other statements of the language.

As already said above, the transfer function we are proposing is just a way to heuristically suggest LU variables and hence it is subject to both false positives and false negatives. We refer the reader to ‘Assessing the Accuracy of LU Oracles’, where we will show and discuss some examples of code chunks leading to false positives and false negatives and we will report on an experimental evaluation meant to assess the accuracy of the LU oracles.

The case of relational analyses. If the target static analysis is based on an abstract domain tracking relational information, such as the domain of convex polyhedra (Cousot & Halbwachs, 1978), then the notion of LU variable no longer corresponds to the notion of unboundedness (as an example, consider the constraint $x=y$). Hence, the definition of the transfer function can be refined accordingly. As an example, when assuming that the domain is able to track linear constraints, a relational version $\mathrm{⟦}\cdot {\mathrm{⟧}}_{\mathrm{r}\mathrm{e}\mathrm{l}}:Stmt\times \mathrm{\wp }(Var)\to \mathrm{\wp }(Var)$ of the transfer function for the assignment statements can be defined as follows:

$$\begin{array}{rl}\mathrm{⟦}x\leftarrow ?{\mathrm{⟧}}_{\mathrm{r}\mathrm{e}\mathrm{l}}(lu)& =\mathrm{⟦}x\leftarrow ?\mathrm{⟧}(lu);\\ \mathrm{⟦}x\leftarrow k{\mathrm{⟧}}_{\mathrm{r}\mathrm{e}\mathrm{l}}(lu)& =\mathrm{⟦}x\leftarrow k\mathrm{⟧}(lu);\\ \mathrm{⟦}x\leftarrow y{\mathrm{⟧}}_{\mathrm{r}\mathrm{e}\mathrm{l}}(lu)& =\{\begin{array}{cc}lu\backslash \{x,y\},\hfill & \mathrm{i}\mathrm{f}x\ne y;\hfill \\ lu,\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e};\hfill \end{array}\\ \mathrm{⟦}x\leftarrow yopz{\mathrm{⟧}}_{\mathrm{r}\mathrm{e}\mathrm{l}}(lu)& =\{\begin{array}{cc}lu\backslash \{x,y,z\},\hfill & \mathrm{i}\mathrm{f}op\in \{+,-\};\hfill \\ \mathrm{⟦}x\leftarrow yopz\mathrm{⟧}(lu)\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e};\hfill \end{array}\\ \mathrm{⟦}x\leftarrow yopk{\mathrm{⟧}}_{\mathrm{r}\mathrm{e}\mathrm{l}}(lu)& =\{\begin{array}{cc}lu\backslash \{x\},\hfill & \mathrm{i}\mathrm{f}op=\mathrm{\%};\hfill \\ lu\backslash \{x,y\},\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}\end{array}$$

Similarly, the relational version for the evaluation of Boolean guards can be defined as follows:

$$\begin{array}{rl}\mathrm{⟦}(x\bowtie y){\mathrm{⟧}}_{\mathrm{r}\mathrm{e}\mathrm{l}}(lu)& =lu\backslash \{x,y\};\\ \mathrm{⟦}(x\bowtie k){\mathrm{⟧}}_{\mathrm{r}\mathrm{e}\mathrm{l}}(lu)& =lu\backslash \{x\}.\end{array}$$

Once again, the definition of ⟦·⟧_rel for the other kinds of statements poses no problem.

The propagation of LU information. Starting from the set $l{u}_{\mathrm{p}\mathrm{r}\mathrm{e}}$ of variables that are LU at the start of a basic block, by applying function ⟦·⟧ (resp., ⟦·⟧_rel) to each statement in the basic block we can easily compute the set $l{u}_{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{t}}$ of LU variables at the end of the basic block. In order to complete the definition of our dataflow analysis we need to specify how this information is propagated through the CFG edges. As a first option we can say that a variable $x$ is LU at the start of a basic block if there exists an edge entering the block along which $x$ is LU; intuitively, if a variable is unconstrained in a program branch, it will be unconstrained even after merging that computational branch with other ones where the variable is constrained. This point of view corresponds to an analysis defined on the usual powerset lattice

$$\mathrm{\exists }\mathrm{L}\mathrm{U}\stackrel{\mathrm{△}}{=}⟨\mathrm{\wp }(Var),\subseteq ,\mathrm{\varnothing },Var,\cap ,\cup ⟩,$$having set inclusion as partial order and set union as join operator. This existential approach may be adequate when our goal is to obtain an aggressive LU oracle, which eagerly flags variables as LU, in particular when adopting the non-relational transfer function.

As an alternative, we can say that a variable $x$ is LU at the start of a basic block only if $x$ is LU along all the edges entering the block; this corresponds to an analysis defined on the dual lattice

$$\mathrm{\forall }\mathrm{L}\mathrm{U}\stackrel{\mathrm{△}}{=}⟨\mathrm{\wp }(Var),\supseteq ,Var,\mathrm{\varnothing },\cup ,\cap ⟩,$$having set intersection as join operator. When using this universal alternative, we will obtain a more conservative LU oracle, in particular when adopting the relational transfer function. Hence, a variable which is constrained in a program branch will remain so even after merging it to other program branches flagging it as unconstrained; while appearing “unnatural” when compared to the previous one, this conservative point of view still makes sense, because the other program branches could be semantically unreachable in all the possible concrete executions.

In both cases, the dataflow fixpoint computation is going to converge after a finite number of iterations, since the two transfer functions are monotone and the two lattices are finite² . In summary, we have obtained four simple LU oracles ( $\mathrm{\exists }\mathrm{L}\mathrm{U},\mathrm{\forall }\mathrm{L}\mathrm{U},\mathrm{\exists }\mathrm{L}{\mathrm{U}}_{\mathrm{r}\mathrm{e}\mathrm{l}},\mathrm{\forall }\mathrm{L}{\mathrm{U}}_{\mathrm{r}\mathrm{e}\mathrm{l}}$) that, to some extent, should be able to guess which variables can be forgotten with a limited effect on the precision of the analysis; each of these can be used to guide a program transformation step that simplifies the target analysis, with the goal of improving its efficiency.

The program transformation step

Algorithm 1 describes how the information about LU variables computed by any one of the oracles described before can be exploited to transform the input CFG. Intuitively, the program transformation should instruct the target static analysis to forget those variables that are not worth tracking. To ease exposition and also implementation, our transformation assumes that each program variable is assigned at most once in each basic block; this is not a significant restriction, since in most cases the input CFG satisfies much stronger assumptions, such as SSA form.

For each basic block $bb\in N$, we retrieve the corresponding set $lu=\mathrm{L}{\mathrm{U}}_{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{t}}(bb)$ of program variables that, according to the chosen oracle, are LU at the exit of the basic block; then, each “assignment-like” statement $(x\leftarrow expr)$ in $bb$ having as target a numeric variable $x\in lu$ is replaced with the nondeterministic assignment $(x\leftarrow ?)$. In the current implementation of this transformation step, the set of “assignment-like” statements also includes conditional assignments, having the form $x\leftarrow (cond?exp{r}_1:exp{r}_2)$; these statements were not considered as potential targets of the havoc transformation in Arceri, Dolcetti & Zaffanella (2023b).

We now provide an example simulating the LU variable analysis and transformation steps on a simple portion of code, focusing on the $\mathrm{\exists }\mathrm{L}\mathrm{U}$ and $\mathrm{\forall }\mathrm{L}\mathrm{U}$ oracles, i.e., the non-relational case.

We conclude this section by clarifying the soundness relation existing between the results obtained by the target numeric static analysis, computing on the numeric abstract domain A, and the concrete semantics of the program, computing on the concrete domain C. Given an input CFG $G=⟨N,E⟩$, the concrete semantics computes a map $Inv:N\to C$ associating a concrete invariant to each node of G. Let $In{v}_A:N\to A$ be the map of abstract invariants for G computed using abstract domain A; then, since the numeric static analysis is known to be sound, for each node $n\in N$ we have

$$Inv(n)⊑{\gamma }_A(In{v}_A(n)).$$

For an arbitrary LU variables map ${\mathrm{L}\mathrm{U}}_{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{t}}:N\to \mathrm{\wp }(Var)$, let $G^{\mathrm{\prime }}=⟨N^{\mathrm{\prime }},E^{\mathrm{\prime }}⟩$ be the CFG produced by Algorithm 1 when applied to G and ${\mathrm{L}\mathrm{U}}_{\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{t}}$. Notice that the algorithm only modifies the contents of the basic blocks of G, without changing its structure; hence, we can easily associate each node $n\in N$ to the corresponding node $n^{\mathrm{\prime }}\in N^{\mathrm{\prime }}$. Let ${Inv}_A^{\mathrm{\prime }}:N^{\mathrm{\prime }}\to A$ be the map of abstract invariants for G′ computed using abstract domain A; then, for each node $n\in N$, letting $n^{\mathrm{\prime }}\in N^{\mathrm{\prime }}$ be the corresponding node in G′, we obtain

$$Inv(n)⊑{\gamma }_A({Inv}_A^{\mathrm{\prime }}(n^{\mathrm{\prime }})),$$meaning that the program transformation is sound when applied to a numeric static analysis no matter the accuracy of the oracle used.

Note that, in general, the stronger result $In{v}_A(n){⊑}_A{Inv}_A^{\mathrm{\prime }}(n^{\mathrm{\prime }})$ does not necessarily hold; this is due to the potential application of non-monotonic abstract semantics operators (e.g., widening). It is also worth stressing that a program transformation such as the one used in Algorithm 1 is sound if the target static analysis is meant to compute an over-approximation of the values of program variables; the transformation may be unsound in more general contexts, e.g., a liveness analysis (Cousot, 2019b).

The static analysis pipeline

We now describe the steps of the overall analysis process, which are summarized in Fig. 2. In the figure, the processing steps are label from Step A to Step H; for each processing step, a directed edge connects its input to its output; blue edges correspond to the processing steps of the original analysis pipeline; red edges highlight the steps that differentiate the modified analysis pipeline from the original one; gray edges, which do not really belong to the static analysis pipeline, correspond to those processing steps that have been added to compare the precision of the target analysis and to assess the accuracy of the LU oracles.

• Step A 

  The input program under analysis is parsed by Clang/LLVM, producing the corresponding LLVM bitcode representation which is then fed as input to clam-pp, the Clam preprocessor component. By default, clam-pp applies a few program transformations, such as the lowering of switch statements into chains of conditional branches; more importantly, in our experiments we systematically enabled the inlining of known function calls, so as to improve the call context sensitivity of the analysis when performing an intra-procedural analysis. Note that Clam/Crab also supports inter-procedural analyses: these are typically faster than full inlining, but quite often produce less precise results.

• Step B 

  The Clam component translates the LLVM bitcode representation into CrabIR, which is an intermediate representation specifically designed for static analysis. In this translation phase a few program constructs that the analysis is unable to model correctly and precisely, e.g., calls to unknown external functions, are replaced by (sequences of) havoc statements. This phase also removes all LLVM phi-instructions, which implement the $\varphi$-nodes of the SSA form, replacing them with additional basic blocks containing assignment statements.

• Step C 

  The havoc analysis computes the set of program variables that are likely unconstrained at the exit of each basic block of the CrabIR representation. As discussed in ‘Detecting Likely Unconstrained Variables’, this step is not subject to a strict safety requirement and hence, in principle, its implementation could be based on any reasonable heuristics; we model it as a classical static analysis and we implemented it by using the Crab component itself.

• Step D 

  This step performs the havoc transformation, using the results of the analysis of Step C to rewrite the CrabIR representation produced by Step B; this is implemented as a simple visitor of the CrabIR CFG, corresponding to Algorithm 1, replacing assignment statements with havoc statements.

• Step E 

  The final processing step is the target static analysis, which reuses the Crab component to compute an over-approximation of the semantics of the havocked CrabIR representation produced by Step D, using the target abstract domain chosen at configuration time. The invariants computed are stored and made available to the post-analysis processing phases (assertion checks, program annotations, etc.).

• Step F 

  In contrast, when the havoc analysis is disabled (i.e., when the analysis toolchain is used without modification), the target static analysis is computed as described in Step E above, but starting from the original CrabIR representation produced by Step B.

• Step G 

  The target-analysis loop invariants produced in Step E and Step F are systematically compared for precision using the clam-diff tool.

• Steps H1/H2 

  The havoc analysis and transformation steps are checked for accuracy by comparing the havoc invariants produced in Step C with the ground truth target-analysis invariants produced in Step F; the details of this accuracy assessment are described in ‘Assessing the Accuracy of LU Oracles’.

As target analyses we will consider the classical numerical analyses based on the abstract domain of intervals (Cousot & Cousot, 1977), for the non-relational case, and the abstract domain of convex polyhedra (Cousot & Halbwachs, 1978), for the relational case. Namely, for the domain of intervals we adopt the implementation that is built-in in Crab, whereas for the domain of convex polyhedra we opt for the implementation provided by the PPLite library (Becchi & Zaffanella, 2018a, 2018b, 2020), which is accessible in Crab via the generic Apron interface (Jeannet & Miné, 2009). Note that we are considering the Cartesian factored variant (Halbwachs, Merchat & Gonnord, 2006; Singh, Püschel & Vechev, 2017) of this domain, which greatly improves the efficiency of the classical polyhedral analysis by dynamically computing optimal variable packs, thereby incurring no precision loss. A recent experimental evaluation (Arceri, Dolcetti & Zaffanella, 2023a) has shown that the PPLite’s implementation of this domain is competitive with the one provided by ELINA (Singh, Püschel & Vechev, 2017), which is considered state-of-the-art.

The experiments described in the article have been run on a few machines, using different operating systems; the experiments meant to evaluate efficiency have been run on a ‘MacBookPro18,3’, with an Apple M1 Pro CPU and 16 GB of RAM; in our configuration the analysis pipeline is sequential, making no use of multi-threading and/or process-level parallelism.

The benchmarks for the experimental evaluation

The preliminary experimental evaluation conducted in Arceri, Dolcetti & Zaffanella (2023b), using the first prototype of the havoc analysis and transformation steps, initially considered the C source files distributed with PAGAI (Henry, Monniaux & Moy, 2012), which are variants of tests taken from the SNU real-time benchmark suite for WCET (worst-case execution time) analysis. Most of these benchmarks are synthetic ones: they were probably meant to assess the robustness of a WCET analysis tool when handling a variety of control flow constructs. Hence, the experimental evaluation in Arceri, Dolcetti & Zaffanella (2023b) extended the set of tests by also considering 10 Linux drivers from the SV-COMP repository (https://github.com/sosy-lab/sv-benchmarks/tree/master/c/ldv-linux-4.2-rc1): being (minor variants of) real code, these benchmarks are typically larger and arguably more interesting than the synthetic ones. For this reason, here we avoid repeating the experimental evaluation for the PAGAI benchmarks, referring the interested reader to Arceri, Dolcetti & Zaffanella (2023b); rather, we extend the set of the more meaningful benchmarks by considering a larger number of Linux drivers. The reader is warned that, since the improved havoc analysis and transformation steps are not fully equivalent to the initial prototype of Arceri, Dolcetti & Zaffanella (2023b), little variations with respect to the experimental results shown in Arceri, Dolcetti & Zaffanella (2023b) should be expected.

In Table 12 (shown in the Appendix) we provide a few metrics for each Linux driver, whose source code is stored in a single file and hence gets translated into a single LLVM bitcode module. The first column associates a unique identifier (‘d $nn$’) to each of the 30 Linux drivers, whose shortened names are shown in the second column; the next eight columns provide the values for the considered metrics. For convenience, the detailed information in Table 12 is summarized in Table 1: namely, for each of the eight columns of Table 12, we report in the rows for Table 1 its arithmetic mean and standard deviation, as well as the most commonly used percentiles (minimum value, first quartile, median, third quartile and maximum value)³ . The reader is warned that, from now on, we will refer to this kind of statistics for what they are, i.e., descriptive statistics for the considered set of benchmarks; no generalization or extrapolation to different contexts is meant.

The four rightmost columns of Table 1 show the number of functions (‘fun’), numeric variables (‘var’), CFG basic blocks (‘node’) and elementary statements (‘stmt’) of the “original CrabIR representation”, as obtained after Step B of the Clam/Crab pipeline. In the CrabIR representation, the elementary statements (‘stmt’) do not include the basic block terminators, i.e., those statements such as ‘goto’ and ‘return’ encoding the edges of the CFG of each function; hence, it may happen that a CFG has more nodes than statements. More importantly, since our LU oracles are only targeting numeric information, when counting program variables (‘var’) we only considered those having a numeric datatype, i.e., those that in the CrabIR representation have a scalar integer or real type; hence, we disregard Boolean variables, aggregate type variables (i.e., arrays) and references to memory regions (i.e., pointers). As a matter of fact, floating point variables are ignored too, because the Crab static analyzer does not fully support this datatype; as a result, even though in principle nothing would prevent our transformation to be applied to floating point variables, in the considered experimental setting it only targets integer ones.

As already noted, this program representation is the result of several transformations, including the LLVM function inlining pass computed in Step A. To better highlight the practical impact of this pass, on the left of the rightmost four columns described above we show the results that would have been obtained, for the same set of metrics, when disabling the function inlining step (i.e., when invoking the tool clam.py without passing option ––inline). Clearly, this program transformation deeply affects both the precision and the efficiency of any numeric analysis: on the one hand, function inlining tends to generate bigger functions, having more basic blocks and variables and hence allowing for the propagation of more contextual information; on the other hand, the same transformation allows for removing the code of all the (already inlined) auxiliary functions having internal linkage, which do not need to be analyzed in isolation.

The impact on code of the havoc transformation

Table 2 summarizes, by means of the usual descriptive statistics, the effects on the CrabIR representation of the 4 havoc transformation variants; the details for each driver can be found in Table 13 in the Appendix. The second column of the table (‘check’) reports on the number of checks performed during the havocking process, i.e., the number of executions of the conditional test $x\in lu$ in line 4 of Algorithm 1. This number, corresponding to the maximum number of statements that could potentially be havocked, is smaller than the overall number of statements of each driver (reported in the last column of Table 1), because as said before the havoc transformation pass only considers the assignment-like statements that operate on variables having a numeric datatype. The following four columns reports on the percentage of assignment-like statements that are actually havocked when adopting each variant of the havoc analysis, i.e., the percentage of executions where the test $x\in lu$ in line 4 of Algorithm 1 is satisfied.

When looking at the data as summarized in Table 2 we observe that the overall effect of the havoc transformation varies significantly depending on the oracle variant chosen. When comparing the percentages of statements havocked by the four oracle variants, going from left to right, we see that the oracle variants become more and more conservative, confirming the intuitive expectations:

• the relational oracles are more conservative than the non-relational ones;

• the universal oracles are more conservative than the existential ones;

• the choice between non-relational and relational oracles has a much greater impact than the choice between existential and universal variants.

Roughly speaking, on the considered benchmarks the transformations based on the relational variants seem really conservative; in contrast, the non-relational ones look rather aggressive, potentially leading to much more significant effects on the precision and efficiency of the target analysis. Observing the percentile values it can be seen that the effect of the havoc transformation step also varies significantly depending on the considered test. By looking at the detailed data in Table 13, we see that almost all tests are affected by the havoc transformation, the only exceptions being drivers ‘d03’ and ‘d14’ when considering the relational oracles: these two drivers are among those having the smallest number of havoc checks (i.e., havocable statements); also, even for these small tests the non-relational oracles are able to havoc a significant fraction of the statements.

The percentage of havocked statements⁴ computed on all tests (see the last line in Table 13) is 78.32% for the non-relational oracle $\mathrm{\exists }\mathrm{L}\mathrm{U}$, while being 10.52% for the relational oracle ${\mathrm{\exists }\mathrm{L}\mathrm{U}}_{\mathrm{r}\mathrm{e}\mathrm{l}}$; the percentages for the corresponding universal oracles $\mathrm{\forall }\mathrm{L}\mathrm{U}$ and $\mathrm{\forall }{\mathrm{L}\mathrm{U}}_{\mathrm{r}\mathrm{e}\mathrm{l}}$ are slightly smaller (69.04% and 10.07%, respectively).

Regarding efficiency (i.e., the computation overhead caused by the havoc processing step in the overall static analysis pipeline), in Table 13 we report, for each driver and each oracle variant, the time spent in the havoc processing steps (Step C and Step D of Fig. 2). We observe that for most of the drivers the time consumed by these processing steps is reasonably small, with a few notable exceptions: in particular, the time spent in the havocking process for ‘d01’ is greater than the time spent on all the other 29 drivers. It should be stressed that most of the processing time is spent in the havoc analysis phase (Step C); only a tiny fraction of the havoc processing time is spent on the havoc transformation phase (Step D). We note in passing that the havoc analysis based on the universal LU oracles happens to be slower. This has to be expected, by the following reasoning:

• the efficiency of the havoc analysis is (negatively) correlated with the size of the sets of variables that need to be propagated;

• hence, to improve efficiency, at the implementation level the havoc analysis represents and propagates the complement $Var\backslash lu$ of the set of LU variables (i.e., the set of constrained variables), because it is typically much smaller than $lu$;

• the universal oracles, being more conservative than the relational ones, compute smaller $lu$ sets, which ends up computing and propagating larger set complements.

Qualitative accuracy assessment

As briefly mentioned before, our LU oracles are not based on Abstract Interpretation theory and, in particular, the semantic transfer functions sketched in ‘Detecting Likely Unconstrained Variables’ are not meant to be formally correct: as a result, when classifying the program variables as constrained or unconstrained, they are subject to both false positives and false negatives.

For ease of exposition, in the following we will mainly refer to the existential variant of the non-relational LU oracle ( $\mathrm{\exists }\mathrm{L}\mathrm{U}$), matched by the corresponding target analysis using the domain of intervals; the reasoning is easily adapted to apply to the other variants.

As discussed before, for each program point $p$, the LU oracle computes the set of likely unconstrained variables, thereby predicting numeric variable $x\in Var$ to be unconstrained when $x\in lu$. The results of the target numerical static analysis using the domain of intervals (on the unmodified program) is used as ground truth: variable $x\in Var$ is unconstrained at program point $p$ if the invariant computed at $p$ states that $x\in [-\mathrm{\infty },+\mathrm{\infty }]$; variable $x$ is constrained at program point $p$ if the computed invariant at $p$ states that $x\in [lb,ub]$, where at least one of $lb$ and $ub$ is a finite bound; as a special case, when the target analysis computes the bottom element $\mathrm{\perp }\in \mathrm{I}\mathrm{t}\mathrm{v}$, i.e., when it flags the program point $p$ as unreachable code, then all variables in $Var$ are constrained in $p$. Therefore, these are the possible outcomes of the LU variable classification phase:

• True Positive (TP): the oracle predicted $x\in lu$ and the target analysis is computing a program invariant where $x$ is indeed unconstrained;

• True Negative (TN): the oracle predicted $x\notin lu$ and the target analysis is computing an invariant where $x$ is indeed constrained;

• False Positive (FP): the oracle predicted $x\in lu$ but the target analysis is computing an invariant where $x$ is constrained;

• False Negative (FN): the oracle predicted $x\notin lu$ but the target analysis is computing an invariant where $x$ is unconstrained.

A too aggressive LU oracle, characterized by a high percentage of FP, will likely cause more precision losses in the target numerical analysis, possibly improving its efficiency; in contrast, a too conservative LU oracle, characterized by a high percentage of FN, will preserve the precision of the target analysis, while also having a limited impact on its efficiency.

In Fig. 3 we provide examples of C-like code witnessing that all the classification outcomes are indeed possible when using the non-relational LU oracles. In these examples, all variables are of integer type and a nondeterministic assignment such as ‘ $x\leftarrow ?$’ is simulated by x = nondet(), where the function call in the right-hand side returns each time an arbitrary integer value. Note that the comments are not meant to show the complete invariant computed when using the interval domain; rather, we only show the portion of the invariant which is relevant to compute the interval value for variable $x$ at the end of each chunk of code.

The examples of TP and TN shown in Figs. 3A and 3B need no explanation and are only provided for the sake of completeness.

False Positives. The FP example in Fig. 3C is rather involved and deserves an explanation. In principle, a FP could be obtained more easily by processing the simpler code in Fig. 3D: here, since in line 5 we have $y\in lu$, the transfer function of Eq. (4) will also flag $x$ as LU, while the interval analysis is able to detect a multiplication by zero and hence conclude that $x$ is actually constrained. However, this simple code pattern cannot be observed in the context of our experimental evaluation because, during Step A of the Clam/Crab toolchain, the last assignment x = y * z is automatically simplified to the constant assignment x = 0: the simplification is triggered by the InstCombine LLVM bitcode transform pass (https://llvm.org/docs/Passes.html#instcombine-combine-redundant-instructions), which is invoked by the Clam preprocessor clam-pp. To avoid this, lines 1–6 of Fig. 3C implement a “mildly obfuscated” assignment of the constant value 0 to variable $z$: on the one hand, the code is complicated enough to confuse the InstCombine transformation step; on the other hand, the code is simple enough to be precisely modeled by the static analysis using the domain of intervals.

The observation above has significant consequences from a practical point of view: since the code we are analyzing is the result of some program transformation/simplification steps, there are code patterns that will never occur in practice and hence can be simply ignored when defining the transfer functions of our LU oracles, without affecting their accuracy. For instance, we could simplify the transfer function in Eq. (5), by ignoring the special case $x\leftarrow y\ast 0$; similarly, we could simplify the transfer function in Eq. (4) by ignoring the special case $x\leftarrow y-y$. This is also the reason why our LU oracles, by design, are unable to distinguish between reachable and unreachable code: in principle, one could lift the abstract domain by adding a distinguished bottom element to encode unreachability and then complicate the transfer function of Eq. (5) so as to detect the special cases $x\leftarrow y/0$ and $x\leftarrow y\mathrm{\%}0$, which are triggering definite division by zero errors. However, these cases too are going to be simplified away in Step A of our toolchain and hence they never occur in our experimental evaluation. Clearly, the design choices above should be reconsidered when targeting different static analysis toolchains that do not apply the above mentioned program transformation/simplification steps.

In Fig. 3E we can see another example of FP, but having a different root cause: intuitively, the static analysis using the domain of intervals can sometimes flag an execution path as unreachable, whereas this precision level cannot be achieved by the LU oracle. In lines 1–4 of Fig. 3E we see again the code pattern to obtain an obfuscated assignment of value zero to variable $z$; knowing that $z\in [0,0]$, the interval analysis can flag the else branch in lines 8–10 as unreachable code, whereas this code is reachable for the LU oracle. Hence, we obtain a FP for $x$ at line 10, where the oracle would predict $x\in lu$; this FP is also propagated to line 11 where, by merging the invariants computed in the two branches of the if-then-else construct, the $\mathrm{\exists }\mathrm{L}\mathrm{U}$ oracle will still predict $x\in lu$, even though $x$ is actually constrained. Note that, in this example, the universal variant $\mathrm{\forall }\mathrm{L}\mathrm{U}$ of our oracle would correctly flag $x$ as constrained at line 11: however, this is only happening by chance; in general, the $\mathrm{\forall }\mathrm{L}\mathrm{U}$ oracle is more prone at generating a FN.

False Negatives. The code in Fig. 3F shows an example of FN: when reaching line 7, we have both $n\notin lu$ and $p\notin lu$ so that, by following Eq. (4), our LU oracle will conclude that $x$ is constrained at line 8; however, this is not the case because in the interval domain we obtain $[-\mathrm{\infty },0]+[0,+\mathrm{\infty }]=[-\mathrm{\infty },+\mathrm{\infty }]$. The code in Fig. 3G shows that another FN can be similarly obtained due to control flow. Here, variable $x$ is unconstrained on line 2; then, the abstract evaluation of the conditional guard (x >= 0) on line 3 leads to using twice Eq. (7) to process predicates $(x\ge 0)$ and $(x<0)$, so that $x$ becomes constrained in both branches of the if-then-else statement (lines 4 and 7); when later merging the invariants of the two computational branches, the $\mathrm{\exists }\mathrm{L}\mathrm{U}$ oracle will still report $x$ as constrained (i.e., $x\notin lu$ on line 9), but this is not the case because in the interval domain we obtain $[-\mathrm{\infty },-1]\bigsqcup [0,+\mathrm{\infty }]=[-\mathrm{\infty },+\mathrm{\infty }]$.

These two FN examples could be avoided by modifying our LU oracles to track more information⁵ . Namely, the oracle could track separately the constrainedness of the lower and upper interval bounds, so as to distinguish for each variable the four cases: $x\in [lb,ub]$ (i.e., $x$ has both a lower and an upper bound); $x\in [lb,+\mathrm{\infty }]$ (i.e., $x$ has only the lower bound); $x\in [-\mathrm{\infty },ub]$ (i.e., $x$ has only the upper bound); and $x\in [-\mathrm{\infty },+\mathrm{\infty }]$ (i.e., $x$ is unconstrained). While interesting, such an approach seems to be tailored to the case of the non-relational LU oracles; we also conjecture that, in general, it would result in a limited gain of precision, while somehow complicating the definition of the transfer functions of the LU oracle.

The code in Fig. 3H shows another example of FN which is caused by the use of the widening operator on the interval domain. When entering the loop on line 3, variable $x$ has value zero, so that $x\notin lu$; in the loop body, variable $x$ is nondeterministically incremented or decremented, so that it remains constrained; hence, the LU oracle quickly converges to a fixpoint where $x\notin lu$. In contrast, when using the interval domain, the loop is entered knowing that $x\in [0,0]$ and each iteration of the loop body will ideally compute a new interval along the infinite ascending chain

$$[0,0]⊑[-1,1]⊑\dots ⊑[-n,n]⊑[-(n+1),n+1]⊑\dots$$

To avoid nontermination, the static analysis tool will resort to the use of the interval widening operator, thereby reaching the loop fixpoint invariant after computing

$$[-n,n]▽[-(n+1),n+1]=[-\mathrm{\infty },+\mathrm{\infty }].$$Hence we obtain a FN, because the oracle predicts $x\notin lu$ whereas the target analysis actually computes $x\in [-\mathrm{\infty },+\mathrm{\infty }]$; this loop invariant is propagated to line 9.

This example of a FN that is caused by the use of widenings in the target analysis is quite interesting. Roughly speaking, such a loss of accuracy of the LU oracle is to be expected because widenings were not taken into account when defining its transfer functions. In principle, by also considering widenings, a modified LU oracle could more faithfully mimic the actual static analysis computation by also guessing when the use of widening operators would eventually cause a constrained variable to become unconstrained. This approach could be interpreted as some sort of dual of loop acceleration (Gonnord & Schrammel, 2014): the goal of loop acceleration techniques, which target loop constructs satisfying some specific properties, is to enhance precision by computing the exact abstract fixpoint of the loop via some algebraic manipulation of the loop body transfer function; hence, Kleene iteration is avoided and, in particular, the precision losses caused by widenings are avoided. Reasoning dually, a new LU oracle could be designed to identify those loops where a huge precision loss is likely to occur, possibly due to the use of widenings: hence, even in this case Kleene iteration would be avoided, but this time with the goal of accelerating the unavoidable precision losses. The main problem with such an approach is that the effects of widening operators are far from being easily predictable. To start with, these operators are typically non-monotonic (Cousot & Cousot, 1977). Moreover, since by definition widenings are not meant to satisfy a best precision requirement, an abstract domain may be provided with several, different widening operators: for instance, in their implementations of the domain of convex polyhedra, libraries PPL (Bagnara, Hill & Zaffanella, 2008) and PPLite (Becchi & Zaffanella, 2020) provide both the standard widening of Halbwachs (1979) and the improved widening operator proposed in Bagnara et al. (2003). On top of this, widening operators are often combined with auxiliary techniques that are meant to throttle their precision/efficiency trade-off: an incomplete list includes the use of descending iterations with narrowing (Cousot & Cousot, 1977), delayed widening (Cousot & Cousot, 1992), widening up-to (Halbwachs, Proy & Raymond, 1994), widening with thresholds (Blanchet et al., 2003), lookahead widening (Gopan & Reps, 2006) and intertwining widening and narrowing (Amato et al., 2016). The choice of a specific combination of techniques is typically left to the configuration phase of the static analysis tool: as an example, the fixpoint approximation engine implemented in Clam/Crab intertwines widening and narrowing, also delaying the application of widenings for a few iterations; widening thresholds are also supported, but these were not used in our experimental evaluation. To summarize, any prediction of the effects of widenings is bound to be strongly coupled with the specific widening implementation and configuration adopted in the considered static analysis tool.

Quantitative accuracy assessment for havoc analysis

Having discussed the potential sources of false positives and false negatives from a qualitative point of view, we now focus on a quantitative assessment. We start by checking the accuracy of havoc analysis in isolation, i.e., the accuracy of the invariants produced by the LU oracles independently from their actual use in the havoc transformation phase.

With reference to the Clam/Crab pipeline shown in Fig. 2, in Step H1 we compare the original target analysis invariants produced by Step F (corresponding to the ground truth) with the havoc-analysis invariants produced by Step C. In particular, in the comparison we will use as ground truth the invariants of the interval domain for the non-relational LU oracles, while for the relational LU oracles will be using as ground truth the invariants of the polyhedra domain.

The systematic comparison is implemented by ad-hoc code that we added to the clam executable: for each function in the considered test, we first compute the corresponding set of numeric program variables; then, for each basic block $bb$ in the CFG of the function, the set of unconstrained variables (ground truth) is extracted by the target analysis invariants computed at the end of $bb$; similarly, the set of LU variables (prediction) is extracted from the havoc-analysis invariants. By comparing these sets, the numbers of TP, TN, FP, FN for basic block $bb$ are computed and added to corresponding global counters for the considered test.

The detailed results are shown in the Appendix, in Tables 14 and 15; the corresponding descriptive statistics are shown in Tables 3 and 4. With reference to Table 14, the second column (‘check’) reports the total number of oracle predictions that have been checked for accuracy: this number is high because, as said before, it has been obtained by adding, for each function analyzed in the considered driver, the product of the number of numeric variables of the function by the number of its CFG nodes; as an example, since driver ‘d06’ has a single function after inlining, we can directly use the numbers ‘var’ and ‘node’ from Table 12 to compute

$$\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{c}\mathrm{k}=\mathrm{v}\mathrm{a}\mathrm{r}\times \mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e}=23,493\times 41,614=977,637,702\approx 977.6\mathrm{M}.$$

The next four columns of Table 14 show the percentages of TP, TN, FP and FN obtained when using the existential oracle $\mathrm{\exists }\mathrm{L}\mathrm{U}$; these are followed by another set of four columns, for the universal oracle $\mathrm{\forall }\mathrm{L}\mathrm{U}$.

The descriptive statistics in Table 3 show us that for oracle $\mathrm{\exists }\mathrm{L}\mathrm{U}$ the average percentages of FP and FN are as low as 2.07% and 0.04%, respectively. While interesting, these descriptive statistics are not the most adequate ones for our accuracy assessment. Hence, in Table 5 we show the values obtained for the metrics commonly used for binary classifiers⁶ :

• ‘accuracy’, computed as $\frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}};$

• ‘recall’, computed as $\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}};$

• ‘precision’, computed as $\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}};$

• ‘FPR’ (FP rate), computed as $\frac{\mathrm{F}\mathrm{P}}{\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{N}};$

• ‘FNR’ (FN rate), computed as $\frac{\mathrm{F}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}};$

• ‘F₁’ (balanced F-score), computed as $\frac{2\mathrm{T}\mathrm{P}}{2\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}.$

It can be seen that the existential oracles are characterized by rather good values for all the metrics, with the exception of ‘FPR’, which is really high. When moving to the universal oracles, ‘precision’ is almost unaffected but we observe a significant decrease of ‘accuracy’ and ‘recall’ values, with a consequential effect on ‘F₁’. These are the direct consequences of the increase in the number of FN, which is expected for what we said in our qualitative assessment. Note that the ‘F₁’ score, being balanced, is giving equal importance to FP and FN misclassifications: in our context, a conservative oracle should prefer a FN (i.e., missing an opportunity to improve the efficiency of the target analysis) to a FP (i.e., potentially incurring a precision loss in the target analysis); an aggressive oracle would rather prefer the other way round. Hence, depending on context, one may opt for using the F_β score

$${\mathrm{F}}_{\beta }=\frac{(1+{\beta }^2)\mathrm{T}\mathrm{P}}{(1+{\beta }^2)\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+{\beta }^2\mathrm{F}\mathrm{N}}$$so as to weight more TP or TN depending on the value of $\beta \ne 1$.

In order to explain the high value of ‘FPR’, we performed a further investigation focusing on those few tests having a high percentage of FP: this allowed us to conclude that these high percentages are directly caused by those program points that are detected to be unreachable by the target-analysis (e.g., the interval analysis in the non-relational case, as is the case for lines 8–10 in Fig. 3E). Looking at the details in Table 14, we see that FP values above 10% are only recorded for drivers ‘d09’ (35.76%) and ‘d21’ (13.97%). As an example, the CFG for function ‘main’ in driver ‘d09’ has 11,967 nodes and 19,338 numeric variables; the interval invariant $\mathrm{\perp }\in \mathrm{I}\mathrm{t}\mathrm{v}$ is computed on 4,281 nodes, leading to $19,338\times 4,281=82,785,978$ checks where the ground truth declares the variable to be constrained due to the unreachability of the node. Since the LU oracle is unable to flag these nodes as unreachable, it classifies almost all of these variables as unconstrained, thereby recording 82,742,672 FP (99.5%) and only 43,306 TN (0.05%). If these unreachable nodes were ignored, for the whole driver we would obtain 1,076 FP, so that the FP percentage would drop from 35.76% to less than 0.01%.

Roughly speaking, the quantitative assessment for the havoc analysis is somehow questionable: by checking the prediction of each variable in each basic block, a correct/wrong prediction is recorded even for those variables that never occur in the considered basic block, adding significant noise to the accuracy assessment. For this reason, in the following section we complement our accuracy assessment for the LU oracles by focusing on the predictions that come into play in the havoc transformation phase.

Quantitative accuracy assessment for havoc transformation

As said above, we now turn our attention to the oracle predictionsa that are actually used during the havoc transformation phase. With reference to the Clam/Crab pipeline shown in Fig. 2, in Step H2 we still compare the original target analysis invariants produced by Step F with the havoc-analysis invariants produced by Step C, but now we also take into account the transformation working on the ‘original CrabIR’; namely, we will only check the accuracy of the LU oracle predictions (with respect to the ground truth provided by the target analysis) when executing the conditional test $x\in lu$ in line 4 of Algorithm 1.

The detailed results of this accuracy assessment are shown in the Appendix in Tables 16 and 17, which have the same structure of Tables 14 and 15 discussed in the previous section; the main difference is that now the contents of the second column (‘check’) correspond to the number of havoc tests computed during the execution of Algorithm 1 (which is the same number already reported in Table 13).

As before, in Tables 6 and 7 we provide the usual descriptive statistics for Tables 16 and 17, respectively; also, the binary classification metrics are shown in Table 8. Here we can see that for the ‘accuracy’ and ‘precision’ metrics we obtain values as good as the ones in Table 5; we record lower values for ‘recall’, as a direct consequence of the increase in the percentage of FN (less evident for the $\mathrm{\exists }\mathrm{L}\mathrm{U}$ oracle). As expected, significantly better values are obtained for the ‘FPR’ metrics, specially for the relational oracles; as discussed in the previous section, this is mainly due to the reduced fraction of FP generated by the unreachable CFG nodes. Interestingly, when comparing to Table 5, ‘FNR’ values are significantly increasing for existential oracles and decreasing for universal ones (with corresponding effects on ‘F₁’): in other words, the effects of FN misclassifications are mitigated when focusing on the actual use of the havoc analysis results.

In summary, this accuracy assessment seems to confirm most of the observations we made in ‘The Impact on Code of the Havoc Transformation’ when discussing Table 2, somehow providing an explanation for them:

• the relational oracles, having significantly lower ‘FPR’ and ‘recall’, happen to be more conservative than the non-relational ones;

• the universal oracles, having lower ‘recall’, are more conservative than the existential ones.

The effect of havoc transformation on target analysis precision

We start by evaluating how the havoc transformation affects the precision of the target static analysis: with reference to the Clam/Crab toolchain of Fig. 2, in Step G we systematically compare the ‘original target-analysis invariants’, obtained in Step F by analyzing the original CrabIR representation, with the ‘havocked target-analysis invariants’, obtained in Step E by analyzing the havocked CrabIR representation.

We first discuss the results obtained when using the interval domain in the target static analysis. Table 18 in the Appendix shows the detailed results obtained when comparing the original interval analysis (‘intervals’) with the havocked interval analyses using the non-relational oracles (‘ $\mathrm{\exists }\mathsf{L}\mathsf{U}$-intervals’ and ‘ $\mathrm{\forall }\mathsf{L}\mathsf{U}$-intervals’). In the table, after the test identifier (‘test’), for the original interval analysis we report the number of invariants computed (‘inv’) and the overall size of these invariants when represented as linear constraint systems (‘cs’). Note that we only consider the target invariants computed at widening points⁷ , i.e., at most one invariant for each loop in the CFG. To compute the size of the constraint systems we use a slightly modified version of the clam-diff tool, where we force a minimized constraint representation so as to (a) avoid redundant constraints and (b) maximize the number of linear equalities; after that, if each invariant $i\in \mathrm{I}\mathrm{n}\mathrm{v}$ has $e{q}_i$ equalities and $ine{q}_i$ inequalities, we compute

$$\mathrm{c}\mathrm{s}=\sum _{i\in \mathrm{I}\mathrm{n}\mathrm{v}}\mathrm{c}{\mathrm{s}}_i=\sum _{i\in \mathrm{I}\mathrm{n}\mathrm{v}}(2\cdot e{q}_i+ine{q}_i).$$

The next eight columns are divided in two groups, for the ‘ $\mathrm{\exists }\mathrm{L}\mathrm{U}$-intervals’ and ‘ $\mathrm{\forall }\mathrm{L}\mathrm{U}$-intervals’ analyses. The first two columns (‘eq’ and ‘gt’) of each group report the number of invariants on which the havocked interval analysis computed the same result (‘eq’) and a precision loss (‘gt’) with respect to the original interval analysis; note that the tool clam-diff also provides counters for the case of a precision improvement (‘lt’) and the case of uncomparable precision (‘un’): these have been omitted from our tables because, in all of the experiments, they are always zero⁸ . The next two columns in each group (‘ $\mathrm{\Delta }$ cs’ and ‘ $\mathrm{\%}\mathrm{\Delta }$ cs’) provide a rough quantitative evaluation of the precision losses incurred by the havocked target analysis: column ‘ $\mathrm{\Delta }$ cs’ reports the size decrease for the computed linear constraint systems; column ‘ $\mathrm{\%}\mathrm{\Delta }$ cs’ reports the same information in relative terms, as a percentage of ‘cs’.

As usual, the contents of Table 18 are summarized in Table 9 by the corresponding descriptive statistics. For this specific comparison, it suffices to say that the ‘ $\mathrm{\exists }\mathsf{L}\mathsf{U}$-intervals’ analysis computes the very same invariants for 27 of the 30 tests, the exceptions being ‘d05’, ‘d25’ and ‘d29’; the ‘ $\mathrm{\forall }\mathsf{L}\mathsf{U}$-intervals’ analysis is able to obtain equal precision also for ‘d25’. When a precision loss is recorded, quite often the only difference between the compared invariants is one or two missing linear equalities; hence, the mean values for ‘ $\mathrm{\%}\mathrm{\Delta }$ cs’ are less than 0.5%.

When considering the polyhedra domain, things get more interesting. To start with, the results obtained when comparing the original polyhedra analysis (‘polyhedra’) with the havocked polyhedra analyses using the relational oracles (‘ $\mathrm{\exists }{\mathrm{L}\mathrm{U}}_{\mathrm{r}\mathrm{e}\mathrm{l}}$-polyhedra’ and ‘ $\mathrm{\forall }{\mathsf{L}\mathsf{U}}_{\mathrm{r}\mathrm{e}\mathrm{l}}$-polyhedra’) show no precision difference at all on the 30 tests; hence, we omit the corresponding tables. This is largely due to the fact that, as we observed in the previous sections, the relational oracles are overly conservative. Since the end goal of our oracle-based program transformations is to obtain significant efficiency improvements and, to this end, some precision loss is an acceptable trade-off, we now consider a target static analysis where the relational polyhedra domain is combined with a havoc processing phase guided by the non-relational oracles (‘ $\mathrm{\exists }\mathrm{L}\mathrm{U}$-polyhedra’ and ‘ $\mathrm{\forall }\mathrm{L}\mathrm{U}$-polyhedra’).

The detailed results for these comparisons are shown in Table 19 in the Appendix and summarized in Table 10, whose columns have the same meaning of those in Tables 18 and 9. As expected, for these combinations the results are much more varied. Only three tests (‘d03’, ‘d21’ and ‘d30’) obtain the same precision for all invariants: on average, a precision loss is recorded on more than a half of the computed invariants. However, by observing the percentile values computed for ‘ $\mathrm{\%}\mathrm{\Delta }$ cs’ in Table 10, it can be seen that these precision losses are somehow limited: for instance, the median values are both below 8%. By looking at the last row (‘all’) in Table 19, which reports the values computed on all tests, we can see that the average constraint system size for the ‘polyhedra’ analysis is $\mathrm{c}\mathrm{s}/\mathrm{i}\mathrm{n}\mathrm{v}=\mathrm{449,483}/\mathrm{11,636}\sim 38.6$ and the average constraint system size difference for the ‘ $\mathrm{\exists }\mathrm{L}\mathrm{U}$-polyhedra’ analysis is $\mathrm{\Delta }\mathrm{c}\mathrm{s}/\mathrm{i}\mathrm{n}\mathrm{v}=\mathrm{39,569}/\mathrm{11,636}\sim 3.4$, corresponding to a couple of linear equality constraints; the value is slightly better for the ‘ $\mathrm{\forall }\mathsf{L}\mathsf{U}$-polyhedra’ analysis.

Going beyond the precision comparison summarized in Table 10, an interesting question is whether or not the choice of combining a relational target analysis with a non-relational LU oracle is going to deeply affect the shape of the target invariants that can be computed: roughly speaking, by following the predictions of a non-relational oracle (which is biased towards preserving only the interval constraints) the program transformation could be compromising the ability of the target polyhedra analysis to compute invariants containing non-interval constraints, i.e., those constraints that can only be defined using 2 or more program variables. We have thus classified the constraints computed by the target analyses on the whole benchmark suite⁹ , distinguishing between interval and non-interval constraints:

• for the (baseline) ‘polyhedra’ analysis, 76.15% of the constraints are interval constraints;

• for the ‘ $\mathrm{\exists }\mathrm{L}\mathrm{U}$-polyhedra’ analysis, 83.12% of the constraints are interval constraints;

• for the ‘ $\mathrm{\forall }\mathrm{L}\mathrm{U}$-polyhedra’ analysis, 82.32% of the constraints are interval constraints.

While as expected the percentage of interval constraints is increasing, it can be seen that the havocked polyhedra analyses are still able to compute program invariants containing a significant percentage of inherently relational constraints.

The effect of havoc transformation on target analysis efficiency

For evaluating efficiency, we compare the time spent in the static analysis pipeline in order to obtain the (original or havocked) target analysis invariants.