Colloquia - A new perspective on the replication crisis: Science is not a signal detection problem

Abstract:

The perceived replication crisis and the reforms designed to address it are grounded in the notion that science is a binary signal detection problem. However, contrary to null-hypothesis significance testing (NHST) logic, the magnitude of the underlying effect size for a given experiment is best conceptualized as a random draw from a continuous distribution, not as a random draw from a dichotomous distribution (null vs. alternative). Moreover, continuously distributed effects selected using a p < .05 filter must be inflated, so the fact that they are smaller upon replication (reflecting regression to the mean) is not a reason to automatically sound the alarm. Considered from this perspective, recent replication efforts suggest that most published p < .05 scientific findings are true (i.e., in the right direction), with observed effect sizes that are inflated to varying degrees. We propose that original science is a screening process, one that adopts NHST logic as a useful fiction for selecting true effects that are potentially large enough to be of interest to other scientists. Unlike original science, replication science seeks to precisely measure the underlying effect size associated with an experimental protocol via large-n direct replication, without regard for statistical significance. Registered reports are well suited to (often resource-intensive) direct replications, which should focus preferentially on influential findings and be published regardless of outcome. Conceptual replications play an important but separate role in validating theories. However, because they are part of NHST-based original science, conceptual replications cannot fulfill the self-correction mission of replication science.