All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
Thank you for addressing the remaining comments from the reviewer. We are now all happy with the article, and in particular the improved presentation. The article is now ready for publication!
[# PeerJ Staff Note - this decision was reviewed and approved by Daniel S. Katz, a PeerJ Computer Science Section Editor covering this Section #]
See previous review comments.
I think the authors have responded well to the previous review comments. Readers can now more easily evaluate the strength of the paper by themselves.
See previous review comments.
See previous review comments.
No additional comments.
Please focus on discussing the weaknesses in the methodology. Reviewer 1 gave you good and detailed hints to improve this.
The paper is still pleasant to read. The improved narrative and structure of the paper, and the move/removal of overstated results, further helps the reader focus and appreciate the contributions of the paper. The related, previous comments by me regarding the basic reporting have been acceptably and honestly answered/dismissed through a reasonable effort by the authors.
The authors now position their study in regard to other existing strands of research, implying that their contribution is novel and valuable. Furthermore, the description of the methodology (case study) is now adequately structured.
However, the paper still does not adequately argue in regard to the weaknesses apparent in the methodology. While most studies are far from perfect, it is important that the authors clarify the weaknesses of their study enough to allow readers to (by themselves) decide how much the conclusions can be trusted. That said, I think the changes required to reach and adequate state is relatively small.
Further details are provided in “Additional Comments”.
A) For Your Information: I wonder whether you need to include this statement:
“Applying constraints on the development process was seen as less compatible with agile development, which is why this article focuses on standards in the first group.”
While I understand the comment and appreciate your wish to be transparent, it might be a bit confusing to the average reader (not reading the paper in detail).
B) Major: The paper now clearly states the type of case used, but it does not argue why this case is suitable:
“A second rationale for a single case is when the case is an extreme or a unique case, however, this does not apply in our context, rather our case is representative or typical (third rationale). The industrial partner and the products developed are typical of many embedded system manufacturers, both in the same domain of industrial communication equipment and in other similar domains. Thus, the results taken from this case are assumed to be typical of the experiences of an average embedded system manufacturer.”
While you can of course do a lot to present a strong positive argument, it should be sufficient/easier to simply provide a negative one here. In other words, you can, in Section 6.3, discuss the most obvious conclusions which could be different/invalid/less strong in more extreme situations. Purely as an example, embedded software developers often end up being the only expert in “their” system components or support tools/toolchains. Presumably this would decrease the value of reviews, or at least imply they should be focused on particular problems. (I could offer a few references on this and other possible talking points, but I think you have a few to choose from already.) If you provide such a small discussion, it should be enough to give a reader a feeling for the most important limits of your conclusions.
C) Major: There are rudimentary responses in your text in regard to the validity concerns I raised, but they are not very clear. Please clarify:
C.1) “…sample choice (convenience, but no strong argument for case)” -> See Comment (B), are you missing input from some important stakeholders due to who you picked for the group? (Note that the triangulation that you make use of and e.g., discuss in Section 6.3 does not help in regard to omissions in regard to sample choice.)
C.2) “sample size (small)” -> See Comment (B), are you missing input from some important stakeholders due to how many you interviewed? (Note that the triangulation that you make use of and e.g., discuss in Section 6.3 does not help in regard to omissions in regard to sample size.)
C.3) “not iterating” and “data saturation not being ensured” -> The triangulation is a possible argument here. However, while you refer to checking that there were no more input in the research process, it is unclear if this was the case. E.g. “The focus group ended with a summarizing event, asking the participants if there were any candidates they had expected to be presented but that were missing, and if they could share any other thoughts or ideas regarding the material that had been presented to them.” (Did the participants state there was nothing more to say?)
C.4) “not recording audio/video from the focus group” -> You do not have to record audio/video, and you were not able to in this case. Sure. However, even Krueger states concerns when using notes and memory (it not being suitable for novices, etc.). If you handled these concerns, then state so.
I still think your analysis of the notes are a bit unclear, but if you were indeed doing this together I guess the risk you missed anything should be small.
D) Minor: The mentions of DoDs in Section 6.3 and 6.5 does not connect well with the rest of the text, now that you have moved much of the discussion of DoDs to an Appendix. I suggest you rephrase/reduce these parts.
The reviewers and I found this paper to be interesting with an important new angle on the topic. Yet, we also see several critical issues that I would suggest you to address before the paper can be accepted. You will find detailed points in the reviews. I want to especially emphasise two improvements:
1) Add a more detailed literature analysis and anchor the whole text better in the existing literature.
2) Improve and detail the focus group in terms of the empirical design, analysis and results.
The language in the paper is clear, unambiguous, and professional, coupled to an acceptable paper structure, and well visualized figures and tables. Sufficient references are provided, but the focus of the paper and associated gaps are not clearly stated (neither in the introduction, nor later). Raw data is not shared, but the methodology is qualitative (meaning that this is not necessarily possible due to privacy issues, or even valuable). If the data was aggregated in several steps these could be valuable to share, but is unlikely it was in this study.
Further details are provided in “General Comments”.
The paper is a “case study”, which could be problematic according to the journal scope description. However, with this the authors mean that they are using a “case study methodology”, i.e. not that they are simply writing up a description of a case. The authors still need to describe the method, and motivate the case, better, but I believe it could still be acceptable (if I interpret the scope of the journal correctly). The research questions are well defined, but (as noted previously) should be better motivated in regard to current research gaps. Otherwise, it is difficult to assess whether the research is relevant and meaningful. The methodology section and supplementary material describe the focus group approach in the case study sufficiently. The approach is not strong considering what is described in the supplementary material, but described to a sufficient level to be possible to replicate.
Further details are provided in “General Comments”.
The described “literature review” has low validity, and the associated results are overreported. They would be better reported as a simple table, stating what is essentially a focus group script (there is no credible motivation for dedicating a whole section to this). The paper also contains a focus group study, which is possible to replicate. According to the supplementary material the approach is weak in regard to sample choice (convenience, but no strong argument for case), sample size (small), not iterating, not recording audio/video from the focus group, data saturation not being ensured, no checking of the results by the participants, and an unclear approach to coding and thematic analysis. Subsequent to the results a set of proposed guidelines are put forth. However, they have no strong relationship to the empirical evidence from the investigation, and should be reduced to avoid overstating the results.
Further details are provided in “General Comments”.
A) Minor: The authors state that: “Standards for functional safety often rely on a plan-driven process with predefined phases.”
This is a bit of an overstatement, which the authors aggravate by stating in Section 2: “… where development is a strictly sequential process through predefined phases (Jonsson et al., 2012).”
This is not necessarily true, particularly the part regarding “strictly sequential”. (It might still be a planned, heavy process.) I suggest you decrease the strength of the statement through the caveat “… even if these are not necessarily meant to be conducted in a strictly sequential fashion” (or similar), as well as remove/rephrase further statement on this being “strict”.
B) Major: The authors state that: “Available research on combining agile and plan-driven methods is mostly from the perspective of utilizing agile practices in existing plan-driven processes. There is a research gap with respect to implementing plan-driven practices in agile processes to increase confidence and quality, the goal of this paper is to fill parts of that gap.”
Please include relevant references, be more specific on what parts of the gap you are aiming to close and clarify why this is important. The above is a very broad statement, and both your abstract and the second paragraph of the introduction suggest you are aiming for a narrow contribution. Essentially, a reader will want to have an argument for why “strategies for increased confidence and quality in tools used for automated software testing in non-safety development may be found or created from concepts and strategies related to safety-critical development”. I agree this is probably true, but you do not provide an argument for it.
Presumably, such an argument could be built based on what “typical” test activities/tooling used in industry for non-safety critical software do not provide.
C) Minor: The authors state that: “Popular among safety standards is to describe this sequential flow through the V-model (Asplund, 2014), illustrated in Figure 1.”
This is also a bit of an overstatement, as many standards try to be process agnostic. I suggest you decrease the strength of the statement by referring to these standards as instead being influenced by the thinking behind the V-Model (or similar).
D) Minor: In regard to the paragraph starting with: “Safety Integrity Levels (SILs)…”
I suggest that you add the caveat that the definitions of contents of these levels can be quite different among the standards. (Aerospace is e.g., very different from automotive in this regard.) Although you mention this in Subsection 2.2 in regard to the tools, a reader will probably start to wonder already here.
E) Major: A better reference on case studies by Runeson is his book from 2012. I suggest you take a look at Table 3.1 in Chapter 3 in that book and make sure you are covering the methodology thoroughly in your paper.
F) Major: The authors state: “The studied case in the research is defined as the industrial partner and the products developed. The unit of analysis is defined as the development and maintenance of the test framework, utilized at the industrial partner for the execution of manual and automated tests of produced products.”
However, there is no discussion of whether this is a suitable case and/or unit of analysis. Why should a reader trust the results from the case, and why would not a different case give completely different result? You need to provide relevant and convincing arguments in regard to this.
G) Major: You are not describing or providing enough evidence to suggest you performed a systematic literature review in Subsection 4.1. It is thus impossible to replicate what you have done in regard to reviewing literature, and the scientific value of this is thus low (beyond serving as a background and related work). You should remove this section, or provide more information/evidence to convince a reader it was likely to arrive at comprehensive results.
H) Major: Following up on (H), as the literature review is not systematic, the whole of Section 5 is basically a description of your focus group script. You can put if forth as a result if you want (perhaps as a table referred to from Section 4). However, as the literature review is not systematic you should be very careful not to overstate the results. I suggest you move what you need to Section 2 or Section 3, and put the rest of the text in an appendix.
I) Major: There is no strong foundation for suggesting the guidelines put forth in Section 7. It is unclear why you would want to suggest these guidelines, as you are not testing them. It would thus make more sense to have these guidelines in the start of a subsequent paper which validates them.
As Section 8 is mostly repeating the results, I suggest you remove Section 7 and add a convincing discussion to support the parts of the guidelines you can motivate separately in what is now Section 8. (In other words, be specific and discuss each result instead of proposing a large concept that is only vaguely related to the each result.) If you want to keep the proposed guidelines, then you need to be much clearer on how you arrive to the proposed guidelines (and only the proposed guidelines) based on the results. Currently this is very vague.
The topic and approach angle is interesting!
There could be more literature references as e.g.
- Joint System safety engineering handbook
- Functional safety and proof of compliance
Topics in these documents that are relevant are:
culture, Culture, tool process, updated tool requirements IEC 61508, and Definition of ready/done.
I also would like to know more regarding relevant analysis that should be performed to ensure relevant tests, automatic tests, tool chain approach
The open questions in the Focus group did not result in a clear answer. And as a result of these comments, the Conclusion was not as concrete as I hoped.
See comments on section 1 of this review above. Could do more literature study
Definition of done is strongly linked to the definition of ready (DoR). DoR is so far not mentioned.
The conclusion could be more specific on the pitfalls, disadvantage and benefits.
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.