Just a minor comment on the p-value section of the method:
[statcheck] can miss tests in the text that are not in the correct format
Statcheck works only for statistics correctly reported to APA standards, but 3 of 4 papers are submitted to journals that don't follow APA conventions. So it's not surprising that statcheck would miss them, however that's not (necessarily) to do with correctness, but APAness. A group of us were considering extending statcheck to cover other disciplines' conventions, but the variation in what is considered "correct" is very variable.
First, this is very good work and a clever idea. I commend the authors for looking closer at this.
One suggestion: The authors here are using the R package rpsychi to examine test statistics. Readers here might interpret this use of the package as being generally applicable to all scenarios of ANOVA or similar models. The rpsychi package relies on least squares estimation, i.e. sum of squares, etc, to reconstruct F-tests. Modern analyses, however, now often employ maximum likelihood in a mixed model or generalized model framework which can, in some cases, produce test statistics different from least squares. The rpsychi package will not reproduce those values, even when they are correct. Differences can also occur in mean and standard error values if model based means (e.g. LSMEANS in SAS) are used/reported when there is missing data or covariate adjustment.
Again, the investigation done here seems reasonable and provides a good cautionary story, but at the same time, it should not imply that everything published can be validated. I would suggest a cautionary statement be made above as to the limits of methods like rpsychi and that its use here matches the analyses and assumptions made in the published articles (I am assuming this is true. I have not checked those articles myself).
You can also choose to receive updates via daily or weekly email digests. If you are following multiple preprints then we will send you no more than one email per day or week based on your preferences.
Note: You are now also subscribed to the subject areas of this preprint and will receive updates in the daily or weekly email digests if turned on. You can add specific subject areas through your profile settings.
Usage since published - updated daily