The error-rate estimates that were obtained from the simulations appear to have been computed incorrectly. Those estimates—which are called “Proportion of type I per comparison error rates” in the Figure 3 caption, “Proportion of type I error” in the Figure 3 graphs, and “type I PCER” or simply “type I error” in some other places in the paper—are presumably intended to estimate the *per-comparison...

100 is not typically considered an adequate number of iterations for estimating error rates (or for estimating proportions more generally). And the fact that a particular run of 100 iterations happened to produce a similar result as a particular run of 1000 iterations does not justify using such a small number. In fact, performing several runs of 100 iterations each, without changing any parameter...

There are several problems with the decision diagram in Figure 7 and with the corresponding recommendations in the text.

First of all, the diagram exempts all planned orthogonal comparisons from any adjustment whatsoever (which has no apparent justification; see https://files.eric.ed.gov/fulltext/EJ1083896.pdf). In fact, unadjusted orthogonal tests produce higher experimentwise Type I error r...

There appear to be some problems in Table 2.

For instance, Bonferroni is categorized here as specifically for "parametric situations." But in fact, Bonferroni is well known to be valid for any tests, parametric or not. Indeed, the Bonferroni inequalities are simply basic facts of probability. Consider for example the case of 2 events, each with a .025 probability. Certainly the probability th...

I get the impression that in the second phrase, it should have read 'because parametric tests are still valid for moderately non-normal samples', otherwise the line of reasoning at the start of this paragraph is not really logical.