Distributions of p-values smaller than .05 in Psychology: What is going on?
A peer-reviewed article of this Preprint also exists.
Author and article information
Abstract
Previous studies provided mixed findings on pecularities in p-value distributions in psychology. This paper examined 258,050 test results across 30,710 articles from eight high impact journals to investigate the existence of a peculiar prevalence of p-values just below .05 in the psychological literature, and a potential increase thereof over time. We indeed found evidence for a bump just below .05 in the distribution of exactly reported p-values in the journals Developmental Psychology, Journal of Applied Psychology, and Journal of Personality and Social Psychology, but the bump did not increase over the years and disappeared when using recalculated p-values. We found clear and direct evidence for the QRP ”incorrect rounding of p-value” (John et al., 2012) in all psychology journals. Finally, we also investigated monotonic excess of p-values, an effect of certain QRPs that has been neglected in previous research, and developed two measures to detect this by modeling the distributions of statistically significant p-values. Using simulations and applying the two measures to the retrieved test results, we argue that, although one of the measures suggests the use of QRPs in psychology, it is difficult to draw general conclusions concerning QRPs based on modeling of p-value distributions.
Cite this as
2016. Distributions of p-values smaller than .05 in Psychology: What is going on? PeerJ PrePrints 4:e1642v2 https://doi.org/10.7287/peerj.preprints.1642v2note This preprint is not peer-reviewed. You may wish to reference the subsequent peer-reviewed version of this article.
Author comment
This is an updated version of the preprint. Funding institution and recipients have been added.
Sections
Additional Information
Competing Interests
The authors declare that they have no competing interests.
Author Contributions
Chris HJ Hartgerink conceived and designed the experiments, performed the experiments, analyzed the data, contributed reagents/materials/analysis tools, wrote the paper, prepared figures and/or tables.
Robbie CM van Aert conceived and designed the experiments, performed the experiments, analyzed the data, contributed reagents/materials/analysis tools, wrote the paper, prepared figures and/or tables.
Michèle B Nuijten analyzed the data, reviewed drafts of the paper.
Jelte M. Wicherts reviewed drafts of the paper.
Marcel ALM van Assen conceived and designed the experiments, analyzed the data, contributed reagents/materials/analysis tools, wrote the paper.
Data Deposition
The following information was supplied regarding data availability:
Open Science Framework (OSF), all analysis code available here (runs all the analyses in the paper)
https://osf.io/pvrtx/. Data is directly imported from Nuijten et al (2015) paper in the analysis code (data itself is available from: https://osf.io/gdr4q).
Funding
The preparation of this article was supported by Grants 406-13-050 (Robbie CM van Aert) and 016-125-385 (Jelte M Wicherts) from the Netherlands Organization for Scientific Research (NWO). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
