All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
Both reviewers were very happy with the changes that you made, and I am happy to recommend acceptance of the paper.
[# PeerJ Staff Note - this decision was reviewed and approved by Maurice H. ter Beek, a PeerJ Section Editor covering this Section #]
Thank you for sending the revised version. The authors have integrated many of the comments from previous iterations, both w.r.t. formatting and more substantial questions, and have provided well-justified reasons in cases where they decided not to do so.
I think this paper is now in a very good shape, and I would like to recommend it for acceptance as-is.
All questions form the initial review are resolved.
All questions form the initial review are resolved.
I have reviewed the response letter and the modifications done by the authors in their text, which fully address the minor concerns I had in my first review, and add intersting information about relational abstract domains, offline vs online program transformation, and other minor issues.
The experiments were already thorough, and the discussion on relational domains is a welcome addition
All data were already provided in the first version.
Congratulations. As you can see, the reviewers are very happy with your paper and only have minor comments. When preparing the final version of your paper, please make sure to address the questions posed by the reviewers.
The paper extends a previous work by Arceri et al. [SAS '23] which proposes to speedup numerical static analysis by relying on an oracle to predict variables which are likely unconstrained (i.e., the analysis has no information on them) and then replacing any operations on such variables with non-deterministic assignments.
The authors propose four different oracles, based on whether variables **may** be unconstrained or **must** be unconstrained, and whether a non-relational analysis (boxes) or a relational analysis (polyhedra) is the target.
They implement their approach in the Crab/Clam-AI framework and conduct extensive experimental evaluations.
The paper addresses an important, and not yet completely solved, issue, namely how to make numerical static analyses scale. Compared to the SAS '23 paper, this paper supplies new experiments, and elaborates on the background, making it self-contained and accessible.
The fact that for non-relational analysis very little gain seems to be possible is somewhat disappointing, but it is positive that this fact is made explicit in the paper.
The findings are well-supported by their experiments, and the extensive tables make it easy to follow along.
Some questions that came to mind:
- I would have liked to see a soundness theorem for the transformation somewhere. Yes, replacing `x=e` with `x=?` is obviously sound for value analyses, but not for arbitrary analyses. Such a replacement in a context of a combined analysis, e.g., additionally tracking live variables or races would make it unsound. A soundness statement (however trivial) would be a good place to emphasize this caveat.
- C-like language with data types? What happens upon upcast? short x; int y; y=x; (y should probably not be marked as LU)? Do you consider this?
- Could one do the LU analysis online? (I know you say it's not A^2I, but if I give up on the soundness of the pre-analysis, it sounds very similar. In principle,
I would even have precise information on what is unconstrained at hand when doing it online?)
- For the \existsLU-polyhedra and the \forallLU-polyhedra, it would be interesting to see whether using the non-relational oracle causes the polyhedra to
only have constraints that are boxes or implied by boxes. Did you measure this? If this is the case, it would hint at this combination not actually being
terribly useful, as the same precision could be obtained with just boxes.
- l.23 and throughout the paper: Having the names of authors in the text "Static analyses based on Abstract Interpretation Cousot and Cousot (1997) correctly approximate" but not offset from the rest of the text by `()` or `[]` seems very strange. Maybe this is how PeerJ wants it formatted, but in this case going with \citet and having the citation as part of the sentence may make it easier to understand.
- l.66: "We model our oracle as pre-analyses on the abstract program". Perhaps I misunderstood: Is it not the input of the oracle that is used when abstractly compiling the concrete program to an abstract program? So the analysis should be on the original, concrete, program?
- l.73: "correct program transformation": When is a program transformation correct? Here, the transformed program has a different semantics if I understand correctly. Maybe sound (admits more behavior) would be better than correct here?
- l.137: Maybe worth to also give the condition that the sequence with widening is ultimately stable, and not just that it overapproximates the join?
- l.145: If I have $\plusminus \infty$ as bounds, then it's not finite bounds
- l.187: Why is it only likely that `x+expr` is unconstrained here? When would it be bounded? If `expr` evaluates to `bot`?
- l.228: The definition is correct but a bit odd, as the cases are not exclusive, and it takes a while to understand that $x\not\in LU \land y\not\in LU$ is the base case.
- l.243: For x \leftarrow y, would x and y not need to be **removed** from LU?
- l.297: Why does \forallLU also start with the set of all variables unconstrained? Somehow one would assume it starts with $\emptyset$...
Minor random observations:
- l.56: I think leveraging usually does not have `on`
- l.101: provided IN/BY the Appendix.
- l.124: a_i is not defined here
- l.128: Why call it correct and not sound (I think that is the more standard terminology)
- l.136: Maybe \Nabla_a to be consistent?
- l.148: Maybe just $\bot \sqleq itv$ is easier to read than we adding $\Leftrightarrow \textsf{true}$
- l.279: Spaces missing in "\existsLUand and \forallLUoracles"
- l.454: "in passing"
The paper presents a pre-analysis whose goal is to transform a program in a way that may decrease the precision but improve the efficiency of a subsequent analysis. More precisely, the pre-analysis consists in guessing the variables that will be unconstrained (i.e., set to top) in the subsequent analysis, so that we can replace their assignment by an havoc instruction seting this variable to a non-deterministic value.
The paper presents the pre-analysis and experimental evaluation with all the details. The paper is well-written and easy to read. The comparison with related work is appropriate.
The presented idea of a pre-analysis to alter the program behaviour and change the precision/efficiency ratio is clear, and the thorough experiments allows assessing how well the idea works.
The contents of the tables are described in detail and they all relate to the research questions (efficiency and precision of the pre-analysis, and impact on the efficiency and precision of the target analysis).
The authors are very honest in their description of their findings. Their approach work quite well in terms of precision (the pre-analysis accurately predicts whether the variables will be unconstrained, and the impact on the precision of the target analysis is minor, and there is even no impact at all on the relational case), but it does not lead to noticeable improvements in efficiency, and the authors acknowledge that.
- L166: why is the lattice non-complete if there are both a join and meet operators?
- L187: note that if the operation can overflow, then some information can be retrieved on x
- L223: why y - y and not, for instance, y/y (note that both seem unlikely to appear)
- L243: This works, but I found it non-obvious why [| |]rel would be defined in terms of [| |]. To me they are completely unrelated. Or maybe the idea is that the analysis would be to the product of boxes polyhedra, and thus that there should be less likely unconstrained values in this product than boxes alone?
- For [| x <- y |]: I think the result should be lu \ {x,y}
- L246: maybe some intuition can be useful here; e.g. if a variable is the top interval in a branch, it will also be so in the join, so existential follows this idea? (and universal is more conservative if the oracle is wrong)
- In Fig 3, we use nondet, but the rest of the paper used ? instead.
- Fig 3g seems to be quite usual (testing a variable that is unmodified after the join). Isn't this a significant portion of the false negatives?
- L690: I wonder if the fact that univeral algorithms is more conservative could not be proved by just considering the transfer functions (one is an approximation of the other; and because all transfer functions are monotonic)
- L723: quite often the only difference between the compared invariants is one or two missing linear equalities => as this paragraph is about the interval domain, probably one or two bounds are missing, rather than linear equalities?
- L817: typo: examplesAmato
- Related work:
- Maybe the work of Li and Rival (2017) about the silhouette abstraction, used as an abstraction to speed-up a target analysis could be mentionned
- General question: you chosed to perform a separate pre-analysis that modifies the program. Would it be possible to do this pre-analysis simultaneously with the real analysis, and modify transfer function (worsening them) on the fly, integrating the compilation and abstract interpretation phase in a single analysis pass? I would be interested in learning about the benefits/drawbacks of such a dynamic method, compared to your more static one.
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.