“Simmons, Nelson, & Simonsohn (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.” - Article summary

A false positive refers to an incorrect rejection of the null hypothesis. The decisions a researcher can make during the research process is called the researcher degrees of freedom. Four common degrees of freedom are choosing sample size (1), using covariates (2), choosing among dependent variables (3) and reporting subsets of experimental conditions (4). The researcher degrees of freedom can significantly increase the false positive rate.

There are six guidelines for authors to prevent the increased rate of false positives:

1. Authors must decide the rule for terminating data collection before data collection begins and report this rule in the article.
2. Authors must collect at least 20 observations per cell or else provide a compelling cost-of-data collection justification.
3. Authors must list all variables collected in a study.
4. Authors must report all experimental conditions, including failed manipulations
5. If observations are eliminated, authors must also note what the statistical results are if those observations are included.
6. If an analysis includes a covariate, authors must report the statistical results of analysis without the covariate.

There are four guidelines for reviewers to prevent the increased rate of false positives:

1. Reviewers should ensure that authors follow the requirements.
2. Reviewers should be more tolerant of imperfection in results.
3. Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions.
4. If justifications of data collection or analysis are not compelling, reviewers should require the authors to conduct an exact replication.