Manipulating the alpha level cannot cure significance testing

, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
1 Department of Psychology, New Mexico State University, Las Cruces, U.S.A.
2 Zoological Institute, University of Basel, Basel, Switzerland
3 Centre for Neuroscience Studies, Queens University, Ontario, Canada
4 Faculty of Applied and Exact Sciences, Metropolitan Technological Institute, Medellín, Colombia
5 School of Mathematical & Physical Sciences, University of Newcastle, Callaghan, Australia
6 Department of Mathematics, State University of New York at Geneseo, Geneseo, U.S.A.
7 Faculty of Psychology, University of Barcelona, Barcelona, Spain
8 Department of Psychology, University of New Brunswick, Saint John, Canada
9 Independent Researcher, New York, U.S.A.
10 School of Psychology, Benemérita Universidad Autónoma de Puebla, Puebla, México
11 School of Psychology, Universidad Adolfo Ibáñez, Santiago, Chile
12 Oncology Laboratory, CCT CONICET Mendoza, Mendoza, Argentina
13 School of Statistics, National University of Colombia, Medellín, Colombia
14 School of Psychology, University of Ottawa, Ottawa, Canada
15 Department of Health Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
16 Department of Mathematics and Statistics, IIT Kanpur, Kanpur, India
17 Jessenius Faculty of Medicine, Comenius University, Bratislava, Slovakia
18 Department of Psychology, Oklahoma State University, Stillwater, U.S.A.
19 Faculty of Statistics, Saint Thomas University, Bogotá, Colombia
20 Department of Allied Health Sciences, University of Connecticut, Storrs, U.S.A.
21 Simón Bolívar University, Caracas, Venezuela
22 Department of Cognitive Science and Psychology, New Bulgarian University, Sofia, Bulgaria
23 Department of Biochemistry, Microbiology, and Immunology, University of Ottawa, Ottawa, Canada
24 Oikostat GmbH, Ettiswil, Switzerland
25 Department of Education, Yamaguchi University, Yamaguchi, Japan
26 Multimodal Interaction Lab, Leibniz Knowledge Media Research Center, Tübingen, Germany
27 Facultad de Psicología, Universidad Nacional de Mar del Plata, Buenos Aires, Argentina
28 School of Pedagogy, Pontifical Catholic University of Valparaíso, Valparaíso, Chile
29 Department of Medical and Surgical Sciences, “Magna Graecia” University of Catanzaro, Catanzaro, Italy
30 Economics Department, University of Campania “Luigi Vanvitelli”, Capua, Italy
31 Department of Psychosomatic Medicine, University of Basel, Basel, Switzerland
32 Laboratoire de Psychologie et NeuroCognition, Univ. Grenoble Alpes, Grenoble, France
33 Faculty of Economics, Chiang Mai University, Chiang Mai, Thailand
34 Department of Statistics, Universidade Federal de Pernambuco, Recife, Brazil
35 Business School, Massey University, Massey, New Zealand
36 Department of Psychology III, University of Würzburg, Würzburg, Germany
37 School of Psychology, National Autonomous University of Mexico, Mexico City, Mexico
38 Faculty of Engineering, University of Porto, Porto, Portugal
39 Department of Psychology, Universidad del Norte, Barranquilla, Colombia
40 Department of Psychology, University of Basel, Basel, Switzerland
41 Facultad de Ciencias Naturales y Exactas, Universidad de Playa Ancha, Valparaíso, Chile
42 Department of Methods and Statistics, Utrecht University, Utrecht, The Netherlands
43 Centre for Mathematical Morphology, MINES Paristech, Paris, France
44 Department of Mathematical Sciences, New Mexico State University, Las Cruces, U.S.A.
45 Faculty of Arts and Science, Kyushu University, Hakozaki, Japan
46 School of Psychology, The University of Adelaide, Adelaide, Australia
DOI
10.7287/peerj.preprints.3411v3
Published
Accepted
Subject Areas
Science Policy, Statistics
Keywords
P-value, Significance, NHST, Alpha level, Threshold, Publication bias, Winner’s curse, Replicability, Bayes factor, Statistics
Copyright
© 2018 Trafimow et al.
Licence
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ Preprints) and either DOI or URL of the article must be cited.
Cite this article
Trafimow D, Amrhein V, Areshenkoff CN, Barrera-Causil C, Beh EJ, Bilgiç Y, Bono R, Bradley MT, Briggs WM, Cepeda-Freyre HA, Chaigneau SE, Ciocca DR, Carlos Correa J, Cousineau D, de Boer MR, Dhar SS, Dolgov I, Gómez-Benito J, Grendar M, Grice J, Guerrero-Gimenez ME, Gutiérrez A, Huedo-Medina TB, Jaffe K, Janyan A, Karimnezhad A, Korner-Nievergelt F, Kosugi K, Lachmair M, Ledesma R, Limongi R, Liuzza MT, Lombardo R, Marks M, Meinlschmidt G, Nalborczyk L, Nguyen HT, Ospina R, Perezgonzalez JD, Pfister R, Rahona JJ, Rodríguez-Medina DA, Romão X, Ruiz-Fernández S, Suarez I, Tegethoff M, Tejo M, van de Schoot R, Vankov I, Velasco-Forero S, Wang T, Yamada Y, Zoppino FC, Marmolejo-Ramos F. 2018. Manipulating the alpha level cannot cure significance testing. PeerJ Preprints 6:e3411v3

Abstract

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p= .05 to .005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of .05, .01, .005, or anything else, is not acceptable.

Author Comment

Published on 15 May 2018 in Frontiers in Psychology 9, 699, https://doi.org/10.3389/fpsyg.2018.00699