Review History
Assessing the magnitude of changes from protocol to publication—a survey on Cochrane and non-Cochrane Systematic Reviews

Thank you for your constructive responses and revisions. I’m delighted to be able to accept your manuscript and I’m looking forward to your work contributing to the important conversations about departures from protocols. Well done.

I have noted a few minor points that I think you could address as part of the proofing process.

Line 156: I wonder if “we found that 30.1% of clinical study reports (CSRs)” might be confusing as you have defined CSRs before (Line 68) but this matches the initial letters of the three words here. Perhaps just “we found that 30.1% of CSRs showed”.

Line 161: Previously you’d mentioned 80% power (equivalent to beta = 0.20), and I suggest using this here, i.e. “beta= 0.20” rather than “beta= 0.19” even if the sample size would provide 81% power.

Line 171: I wonder if “For a SR to get a high score, no or one non-critical weakness is necessary must be found” should be something like “For a SR to get a high score, no or only one non-critical weakness should be found”?

You normally have spacing between paragraphs but not at Lines 173/174, 176/176, 179/180, 241/242, 285/286, 293/294, 379/380, 384/385, and 388/389.

Lines 180–181: From your rebuttal comment, I wonder if “in a single manner single extraction” should have been “in a single extraction”.

Reference 17 on Line 472 ends with a semicolon. You might find https://support.posit.co/hc/en-us/articles/206212048-Citing-RStudio useful for the citation.

Figure 3 appears a little fuzzy for me (Figure 2 looks fine) and it is possible that a higher-resolution version or uncompressed version will be needed for the final version of your manuscript.

**PeerJ Staff Note:** Although the Academic Editor is happy to accept your article as being scientifically sound, a final check of the manuscript shows that it would benefit from further English editing. Therefore, please identify necessary edits and address these while in proof stage.

Thank you for your revised manuscript and rebuttal. We don’t have additional comments from our reviewers but I will make a few below. These are often quite minor, but there are a few more than I’d like for copyediting and I hope that this will be a useful step in preparing your manuscript. Many of these comments are stylistic and I’ll leave the use of these up to you. Providing these points are all addressed (through changes or rebuttal), and no new points come to my attention, I anticipate quickly recommending acceptance of your revised manuscript.

Lines 38 and 41: Apologies for the pedantry, but could you keep the hyphen (or its absence) consistent for “non[-]CSRs”. See also Lines 47, 48, 51/52, 54, etc. where the hyphen seems consistently included.

Lines 46 and 47: Can you make it clear that the numbers in parentheses are standard deviations? Also, I couldn’t see these values from the abstract in the manuscript itself.

Line 48: I suggest “67.0%” for consistency in decimal places with “54.6%” on the previous line, but given n=97/group, since each n=1 counts for slightly more than 1%, I’d use integer percentages here and throughout the manuscript. See Lines 247, 248, 259 (note integer already on 258), 261, etc., and Table 2, etc.

Line 63: Missing period.

Line 86: Spurious space before period. See also Line 121 and potentially elsewhere.

Line 91: As I’ve also seen the “P” described as “population” (as you refer to on Line 208, albeit not directly as what is meant by the “P” in “PICOS”), and even as “problem”, would it be worth mentioning at least the first alternative here (e.g. “Patients/Population”)?

Line 148: Rather than “randomized” (which suggests random allocation), perhaps “randomly sorted” or “sorted in a random order”? See previous comment and your response regarding Line 134 in the original version of the manuscript.

Line 152: Presumably this refers to the “rnorm()” function (no hyphen, and I suggest appending the parentheses)?

Lines 160–162: Rather than powering on the observed difference, powering on the smallest important difference would be safer. Perhaps, the fact that you were interested in differences smaller than 31.5% could be mentioned here?

Line 164: Perhaps “two-sided” or “two-tailed” would be more familiar to some readers than “bilateral”?

Lines 164–166: I can reproduce the n=97/group here, but only by assuming you were using 0.4 and 0.6 as the proportions (i.e., the worst-case scenario). Slightly fewer would have been needed for other differences of 0.2 which are centred away from 0.5 (e.g., 0.35 versus 0.55, a mean of 0.45 which is closer to your observed data, requires 96/group). Could you clarify that your calculation here is conservative and/or under the worst case for a difference of 0.2?

Lines 172 and 176. I wonder if “found” (as used on Line 178) would be clearer language here than “comprised”.

Lines 181–182: Is there a word missing in “in a single manner single extraction.”?

Line 215: I think I’d find the reference to using Chi-squared tests “to compare the outcomes’ means for the PICOS changes” confusing given that this test compares categorical outcomes and so distributions/proportions rather than means.

Line 251: I’m not sure the word “written” here is needed.

Lines 251–254: Perhaps make these values clear with “...with a mean (SD) duration of...” on Line 251?

Line 264: I don’t think you need the comma in “24 SRs, (24.7%)”. Similar point for Line 275 (“and major changes, 35/108 (32.4%)”—although you could add another comma after this quote to make the values a parenthetical aside) and it might be worth checking commas more generally.

Lines 280–281: Seems to repeat 279–280.

Lines 295–298: The negative wording here makes this harder to follow, I think. In any case, I wonder if a brief aside to make sure the reader remembers why you mean by “magnitude” (Line 295) and perhaps “not reporting” rather than “non-reporting” (Line 297).

Line 342: Perhaps “...CSRs, AND found that 38%...”?

Line 387: “confront” seems a stronger word than necessary. Wouldn’t you have queried them?

Lines 386–388: I’m not convinced that this is a limitation. Your research seems clearly intended to investigate the research question for intervention studies (consider Lines 124–125, 138, 143, 238, and 240) so I wouldn’t consider this a limitation.

Lines 396–397: While I think great caution is required in attributing causality (that non-CSRs are worse because they are non-CSR rather than because of researcher experience, methods experts in the team, funding availability, etc. being associated with the non-CSR/CSR status and causally associated with the outcomes here), at the same time, the comparison is still entirely valid as reflecting the state of affairs. I guess what I’m saying is that I think you could delete the word “great” from Line 396 without over-reaching (in my view).

Lines 403–405: You could, if you wished, point to ChatGPT as having been considered (and presently found wanting) for performing systematic reviews entirely, e.g., Qureshi R, Shaughnessy D, Gill KAR, Robinson KA, Li T, Agai E. Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation? Syst Rev. 2023;12(1):72. doi:10.1186/s13643-023-02243-z

Thank you for your interesting manuscript. We have comments from two reviewers which I invite you to respond to in a revised version of your work. Each of the reviewers’ points should be clearly addressed in your rebuttal—for each explaining either how the manuscript has been changed in response or why no changes have been made. I will ask you do the same to the additional points I’ve included below.

Our two reviewers have collectively raised fairly specific and direct points, all of which should be responded to. I think you would be able to address these without great difficulty, although you could make larger changes in response to some of Reviewer #2’s suggestions.

Line 47: Perhaps related to Reviewer #2’s point about percentages, I wonder how useful the two here are (particularly since they add to 100, making one redundant). Perhaps the mean (or similar) changes per review would be more useful here? I think 108/227 would give 47.6% rather than 47.5 as per the text (with an equivalent discrepancy for 119/227). See also Line 231.

Line 90: While I agree that all are important, the claim that these are equally important is very specific and seems to need some justification. Perhaps a reference, or references, might help here?

Line 134: I think you might mean “randomly sampled from” or “randomly ordered” (as shown in Figure 2, with the sampling taking place later as per Lines 137–138?) rather than “randomized” (i.e., allocated to groups) here. I felt that this was a point Reviewer #2 might have been alluding to as well.

Lines 141 and 145: I think you could explain where the 20% difference hypothesised came from (and why smaller differences would not be of practical importance). Your calculation seems to be for a Chi-squared test (rather than a LR test, say) and to aid reproduction by readers, you could make this explicit.

Line 183: To be pedantic, means and SDs don’t have corresponding CIs (sample sizes would be needed for this), but perhaps you mean the differences in means and their CIs?

Line 184: A Mann-Whitney U test isn’t a test of means, and is only a test of medians in particular circumstances, so it’s unclear to me why you report means and SDs (suggesting that these are readily interpretable, which is seldom the case in my experience outside of certain distributions, e.g. Normal or Poisson) and then use non-parametric tests (suggesting that the assumptions for, say, t-tests were not sufficiently well satisfied). Giving 95% CIs around means, but not giving estimates for the difference in means, along with its own CI, could be justified, but it would be unusual. Given Reviewer #2’s point about potential confounding, quantile regression to model medians would be only possible approach here.

Lines 185–187: I think a good goal for the statistical methods is that a competent (bio)statistician could read them and, if they had access to the raw data, could expect to be able to reproduce the results with confidence. On reading the methods, I don’t think I’d necessarily know exactly what was going to be done here. Could you make this clearer?

Line 189: I’m not sure that I’d describe ARR as a “method” as such. The particular approach used to calculate the 95% CI would be helpful to note though.

Line 193: RStudio is an interface to an R installation (and does no calculations itself), so the version needed here would be the version of R, along with the names and versions of any non-base packages used.

Lines 210–211: The methods talked about means and SDs as descriptives, which would (subject to the reader understanding the nature of the distribution, and other summary statistics might be more useful if the data was not approximately Normally distributed) give the reader some idea of the data. However, CIs are inferential. Are you interested in estimating the time between publications (for which a CI would be useful) or in describing this (for which a mean and SD, or some other combination of summary statistics would be useful)?

Lines 211–213: A similar point would apply here, but a CI along with the point estimate for the difference would be much more useful than a p-value in isolation (Line 213). I appreciate the limitations from MW-U tests, but I wonder if they can be avoided by considering other methods here?

Lines 213–215: Same point again.

[# PeerJ Staff Note: Please ensure that all review and editorial comments are addressed in a response letter and any edits or clarifications mentioned in the letter are also inserted into the revised manuscript where appropriate. #]