Review History
Resampling-based methods for biologists

Thank you for your revisions, which addressed all of the outstanding queries. There are a few small editing points that I’ve picked up when re-reading your manuscript, but these are minor enough for me to leave them for you to address as you see appropriate in the proofing stage. Some of my comments below are entirely stylistic, some are very pedantic, and some depend on possible (mis)readings of the text that might be pushing plausibility a little too much and I’m happy leaving any changes that you make in response to these entirely up to you. Well done on writing what I’m sure will be a very useful, and often referenced, article.

My apologies for not mentioning this earlier, but I wonder if a footnote or parenthetical comment around Line 122 along the lines that “If the reader executes this code, their values are likely to be slightly different due to their random number seed being different from ours. This also highlights the importance of explicitly setting the random number seed if results are to be reproducible.” might not be both reassuring and helpful to readers (most statisticians I know have one, and only one, horror story about needing to reproduce results when they had inadvertently relied on the system clock to provide the random number seed!)

Line 165: There is no space after the plus-minus sign (“…use estimate ±2SE to…”), but on Line 188, there is (“…estimate ± 2BSE…”). The space also appears in Figure 2 (twice) so you may wish to add one to Line 165 for consistency.

Line 173: There is an extra comma after the second “many” in “For example, we can make many, many, copies of our sample data…”

Line 232: While the differences referred to here will probably all be numerically different, this is not guaranteed and you could delete “different” from “Above, we stored the 10,000 different differences in means in a variable named…” without loss (this depends, I think, on exactly how you interpret “different” and whether a clone by chance is indeed “different”).

Lines 264, 287, 315, and 317: The minus signs here seem overly long to me, as if they were em-dashes perhaps? It’s definitely longer than the minus sign on Line 299, for example. My apologies if my eyes are deceiving me here. Looking more carefully, the minus sign back on Line 139 looks like it might be an ordinary hyphen rather than a mathematical symbol. You might prefer to use $\pm 2$ for that instance in any case (as you appear to do on Lines 165 and 188, for example). Similarly, the minus sign on Line 141 looks like an ordinary hyphen. What I believe should be an en-dash (for a range of values) on Line 199 looks like it might also be an ordinary hyphen as well. There may be other instances of these, but I’ll leave it up to you how pedantic you are about your typography.

Line 279: The text here extends into the right margin. See also the output between Lines 297 and 298, and between Lines 397 and 398.

Line 370: Given your pattern of comma use, I would have expected one after “applications” in “For computationally intensive applications it may not be possible…”

Lines 411–412: I wonder if “However, the permutation-based test may also reject the null hypothesis if the population means are the same in the two groups, but the population variances differ.” will be entirely clear to all readers. All hypothesis tests could reject the null given that it is in fact true, irrespective of any assumptions, and we can only hope that this is at the rate described by alpha. Perhaps this sentence could emphasise that the rejection rate could differ from the nominal alpha?

Line 430: You say “…the distribution of residuals is right skewed and, thus, far from Normal…” but right skew in itself, i.e. without any qualification as to the degree of skew, would not justify “far from Normal” (i.e. slight right skew would be close to Normal rather than far from it). Perhaps, “…the distribution of residuals is strongly right skewed and, thus, far from Normal…” or “…the distribution of residuals is right skewed and, thus, non-Normal…” depending on how you interpret Figure 6B.

Line 447: There isn’t any issue with the terminology “among-beach variability”, but, in my experience, “between-beach variability” would be more usual, at least for the hyphenated version (along with “within-beach variability”, c.f. the usual expressions between-subject and within-subject). You use the more normal hyphenated forms on Line 595 (“…both within- and between-cluster variability.”) but then also use among (in a non-hyphenated form) on Line 617 (“…both within and among clusters…”). (You also use “among” in other places, but these were the two instances when it caught my attention.)

Lines 456–458: Percentile methods do attempt to address skew in the sampling distribution. I also wondered if the “…less accurate than…” should be qualified, e.g. “…less accurate on average than…” as bias correction can still worsen coverage.

Line 501: I think I understand why you’ve used theta-hat here, as this is the statistic being used, but this sentence seems also able to be read as referring to “…the true population…[which is] theta-hat”. rather than “…the true population…[estimated by] theta-hat”.

Line 521: You might be doing this intentionally, which is fine, but I wondered about whether the “3” here should be subscript. See also Lines 528 and 530.

Box 1: For the final point, I think an “as” (or similar) might be missing, i.e. “The p-value is the chance of obtaining a sample statistic as extreme AS (or more extreme than) the observed sample statistic, if the null hypothesis is true.”

References: Shouldn’t “pva” on Line 690 be “PVA”? Also for this title, normally you capitalise the first letter after a colon (e.g. Lines 685 and 687) but here you’ve left “effects” with a lower case “e”. Line 745 also has a lower case letter after the colon in an article title, as does Line 770. There might be other instances of this, but generally you do use a capital after a colon in the article title.

[# PeerJ Staff Note - this decision was reviewed and approved by Dezene Huber, a PeerJ Section Editor covering this Section #]

Thank you for your responses and edits. The two original reviewers have looked at your revised manuscript. One has indicated no further suggestions and the other has made some suggestions that ought not to be too difficult to address.

I’ve made some relatively minor additional comments myself below and I would hope that if you can address (through revision or rebuttal) each of the comments from Reviewer #2 and myself, then this will be the final round of revisions for what I think should be a very useful addition to the literature. Please note that several of my comments are stylistic and you should see these as nothing more than suggestions for you to consider.

Lines 156–158: While you might be assuming the n=75 case here (from two lines above), it seems useful to remind the reader that the CLT provides a good approximation to the Normal for sufficiently large sample sizes, where “sufficiently” will depend on the Normality of the population distribution and appears to include n=75 in this particular instance. You already cover this point back on Lines 150–151, where you could add “irrespective of the population distribution”, and again in several subsequent parts of the manuscript, so I’m only suggesting you consider a gentle reminder to the reader of this point here.

Line 311: I wonder if adding “absolute” here might be useful for readers: “the probability of observing aN ABSOLUTE difference in sample means as large or larger”. Again, this is made clear in the following text, so this is just a suggestion for you to consider gently reminding readers here. Related to this, you could consider adding “two-tailed” before “p-value” on Line 313.

Line 324: Rather than “sure”, do you mean “confident”? (As you say earlier on Line 191.)

Line 324: Do you want to keep the decimal places consistent here, i.e. -38.9 and 3.6, or is the use of two significant figures for both your preference?

Line 353: This is a very minor point, but is the switch from 10,000 samples to 1,000 intentional at this point? This seems to be the solitary instance of 1,000 samples in the body of the manuscript.

Line 368: Do you mean “test A statistical hypothesis” or “test statistical hypothesEs” (I’m guessing the latter here)?

Line 390: Since “data” is plural and you refer to “data and R code” on the previous line, perhaps “is” here should be “are”?

Lines 483–510: I wasn’t sure if there was a pattern here as to when you quote “best” (Lines 488, 499, 503, and 504) and when you don’t quote it (Lines 501 and 508).

Line 525: This is purely stylistic, but given your punctuation elsewhere (e.g. the following line), I was expecting a comma after “To some readers” here.

Lines 535–536: I appreciate that you are not trying to describe the history here, but I wonder if adding “extending work on the jackknife, a linear approximation to the bootstrap.” or similar would be useful here, and might be a term that some readers have encountered. You could add a reference to Quenouille (1949) if you really wanted to here.

Line 537: I apologise for another stylistic comment, but is the ordering of words in “can be only as good as the original data” intended or did you want “can only BE as good as the original data” (the latter seems much more common to me, although neither seems wrong)?

Line 599: As a final stylistic comment, with a hyphen in the parenthetical abbreviation “(PIT-)” should there also be one after the “transform” immediately prior?

Figure 5: Do you want to elaborate on the “black horizontal line” as there are two of these in the figure? E.g. “Our observed sample statistic, (\bar{x}_starved - \bar{x}_fed = -18.3, is indicated by the LEFTMOST black horizontal line.”

The reviewers have made some positive comments about your interesting manuscript, including its potential importance and the quality of your writing, and I agree with these. They have also raised some important issues that will need careful thought and potentially some significant reworking of your manuscript. All of their comments, and mine below, should be responded to point-by-point with either changes made to the manuscript or an argument for why no changes were made for each one.

Reviewer #1 makes a very good point about the use of code and I think this comes back to your target audience (I agree with Reviewer #2 that the manuscript is best seen as a “tutorial review”). The importance of this manuscript, for me at least, is introducing these concepts and providing some illustration of how they can be applied and I suspect that for some/many of the likely readers, R will seem a little arcane at first. I agree that if the code-based approach is taken, you need to be sure that potential readers are provided with enough foundations to appreciate the material (both code and output) you present, whether this is through providing more comments, a short tutorial to R, or some other approach. For example, the code between Lines 113 and 115 seem likely to confuse someone without some experience with R (particularly the second line). Their other comments are all very insightful and should also be extremely useful to you in revising your manuscript.

Reviewer #2 makes an also very good point about when these methods do not work well. I think that if the manuscript is to serve as a “tutorial review”, it is important that the reader doesn’t come away with the idea that this solves all of their modelling issues. As this reviewer notes, there are often important assumptions being made and a single best solution is not always accepted by all (bio)statisticians (e.g., resampling for mediation models, with random effects, etc.) and it seems crucial to me that the reader appreciates when they should approach a (bio)statistician or someone else with particular expertise in these methods. As a biostatistician who works with researchers, including from ecology, some of whom like to perform their own statistical analyses, this can be a challenging demarcation to make clear (and it will vary from non-statistician to non-statistician, as well as from (bio)statistician to (bio)statistician). I strongly suspect that a researcher like the ones you describe on Lines 40–43 will still struggle with some of these models when resampling is used. As with Reviewer #1, Reviewer #2’s other comments are all thought provoking and should be extremely useful to you when revising your manuscript. In particular, I think a zip file (either to be hosted on PeerJ or on your own website) containing all data and code together would be extremely beneficial to the reader.

Both reviewers have indicated that they are suggesting some of their own work and you are of course free to use or not use these. In this particular case, I think that their suggestions could be excellent additions to your manuscript and I hope that you will carefully consider each addition they recommend, but decisions around using these are entirely yours.

I will make some additional comments of my own to complement those from the reviews.

Line 8: I wonder if you might think it worthwhile to clarify that when you talk about normality, equal variance (and note that you use “constant variance” on Lines 29, 36, etc. and so might prefer to standardise this terminology), and independence, you’re referring to residuals (to approximate the distribution of the error term) and not to data (c.f. Line 29). I raise this especially as the sentence starts by talking about “Ecological data” and in my experience, this point is not always well appreciated by non-statisticians. If you agree, this could also be made clearer on Line 36.

Line 30: “and mostly rely ON analytical formulas”

Line 31: I wonder if this would be a good place to also mention non-parametric tests, correlation coefficients (including parametric and non-parametric), and even simple descriptive statistics (considering, for example, whether it is the sample mean or the sample median that is of actual interest is, in my experience, often neglected by researchers). These should all be included in an introductory course that extends to linear regression (I assume, but you should clarify, that this is the form of regression that you are intending on Line 28—the description here wouldn’t, for example, match logistic or Poisson regression—as per your reference on Line 33). This is the level I would associate with given your description on Lines 61–63, although those researchers who are entirely self-taught will also be covered by that. I completely agree that the traditional non-parametric approaches will not generalise to address more complicated situations.

Line 32: Given your comments about Chi-squared and F distributions above, and give my experiences with teaching such courses, I’d expect Chi-squared tests, one-way ANOVAs, and Kruskal–Wallis tests to also be covered. Perhaps it is more a matter that such courses often focus on univariable (i.e., one independent variable) and univariate (i.e., a single dependent variable) models rather than one or two sample problems? For a course that covers simple linear regression, I might, depending on the software being used, be tempted to include Huber-White standard errors and perhaps even clustered Huber-White standard errors as ways of relaxing some of the standard assumptions (along with unequal variance versions of the t-test and one-way ANOVA, and non-parametric tests for simpler models).

Line 44: Perhaps worth emphasising that the person performing the analyses does not have to perform this integration! The difference in interpretation of GLMMs and GEEs is, I think, often not well understood though and this is perhaps the point you were interested in here?

Line 49: I think you should consider if the challenges of appropriately preserving structure (including but not just clustering) in the data is sufficiently complex to warrant being a “main limitation”.

Line 111: Perhaps use, or at least note, the z-values of ±1.96 for the 95% CI here and/or on Line 127 (and perhaps again on Line 144 if that’s not too pedantic). Otherwise, I feel that some readers will be slightly confused when they look at other references.

Line 118: A reference or two on the CLT might be useful for some readers.

Line 235: Personally, I’d be much, much more cautious about a one-sided test here, only using this if a result in the opposite direction would absolutely not be commented on (even if it would otherwise have been statistically significant) rather than based on a belief that a result in the opposite direction was “unlikely”. Surprising findings, such as the negative correlations within clusters you mention later, shouldn’t be lost just because the researcher committed to a one-sample test.

Lines 252–253: In the presence of randomisation, confounding is not minimised but is rather completely eliminated. Confounding is a form of bias with a confounder being a variable that is associated with both the exposure and the outcome. With random assignment, any observed association between a “confounder” and the exposure can only be by chance and so cannot reflect any bias (systematic departure from the truth). Missing data mechanisms can induce such a bias, but in the absence of some such bias, confounding is not possible.

Lines 260–263: See comments from Reviewer #2. I think that this nicely illustrates the complexities of resampling-based approaches.

Lines 437–439: Of course, rather than modelling random effects, there are many covariance structures that could be imposed on the residuals, including allowing for negative covariances/correlations (although these are rather rarely the case in my experience).

Lines 445–452: I wonder if the segue into Bayesian methods is worthwhile here. While this is a subjective viewpoint, I’m not sure that a reader of the type you described back on Lines 61–63 will gain any real understanding from this brief aside.

I look forward to seeing the revised version of your manuscript.

[# PeerJ Staff Note: It is PeerJ policy that additional references suggested during the peer-review process should only be included if the authors are in agreement that they are relevant and useful #]