False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant - summary of an article by Simmons, Nelson, & Simonsohn (2011)

Critical thinking
Article: Simmons, Nelson, & Simonsohn (2011)
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant


This article is about two things:

  • despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to false find evidence that an effect exists than to correctly find evidence that it does not.
  • a solution to that problem.


One of the most costly errors is a false positive.

  • The incorrect rejection of the null hypothesis.
  • Once they appear in the literature, they are persistent.
    • Because null results have many possible causes, failures to replicate previous findings are never conclusive.
    • Because it is uncommon for prestigious journals to publish null findings or exact replication, researchers have little incentive to even attempt them.
  • False positives waste resources

They inspire investment in fruitless research programs and can lead to ineffective policy changes.

Ambiguity is rampant in empirical research.


As a solution to the flexibility-ambiguity problem, there are offered six requirements for authors and four guidelines for reviewers.

This solution substantially mitigates the problem but imposes only a minimal burden on authors, reviewers, and readers.
Leaves the right and responsibility of identifying the most appropriate way to conduct research in the hands of researchers, requiring only that authors provide appropriately transparent descriptions of their methods so that reviewers and readers can make informed decisions regarding the credibility of their findings.

Requirements for authors

1. Authors must decide the rule for terminating data collection before data collection begins and report this rule in the article.

2. Authors must collect at least 20 observations per cell or else provide a compelling cost-of-data collection justification.
Samples smaller than 20 per cell are not powerful enough to detect most effects.

3. Authors must list all variables collected in a study
Prevents researchers from reporting only a convenient subset of the many measures that were collected, allowing readers and reviewers to easily identify possible researcher degrees of freedom.

4. Authors must report all experimental conditions, including failed manipulations
Prevents authors from selectively choosing only to report the condition comparisons that yield results that are consistent with their hypothesis.

5. If observations are eliminated, authors must also report what the statistical results are if those observations are included.
Makes transparent the extent to which a finding is reliant on the exclusion of observations, puts appropriate pressure on authors to justify the elimination of data, and encourages reviewers to explicitly consider whether such exclusions are warranted.

6. If an analysis includes covariate, authors must report the statistical results of the analysis without the covariate
This makes transparent the extent to which a finding is reliant of the presence of a covariate, puts appropriate pressure on authors to justify the covariate and encourages reviewers to consider whether including is warranted.

Guidelines for reviewers

1. Reviewers should ensure that authors follow the requirements

2. Reviewers should be more tolerant of imperfections in results

3. Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions.

4. If justification of data collection or analysis are not compelling, reviewers should require the authors to conduct an exact replication.

General discussion


Criticism of the solution comes in two varieties:

  • it does not go far enough
  • it goes to far

Not far enough

The solution does not lead tot the disclosure of all degrees of freedom.

  • it cannot reveal those arising from reporting only experiments that ‘work’

Authors have tremendous disincentives to disclose exploited researcher degrees of freedom.

To far

the guidelines prevent researchers from conducting exploratory research.


Solutions rejected by the authors for they are less practical, less effective or both.

  • Correcting alpha levels
  • Using Bayesian statistics
  • Conceptual replications
  • Posting materials and data
Connect & Continue
WorldSupporter Resources

WSRt, critical thinking - a summary of all articles needed in the second block of second year psychology at the uva


This is a summary of the articles and reading materials that are needed for the second block in the course WSR-t. This course is given to second year psychology students at the Uva. This block is about analysing and evaluating psychological research. The order in which the

Contributions, Comments & Kudos

Add new contribution

This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Enter the characters shown in the image.
Author: SanneA
Join World Supporter
Join World Supporter
Log in or create your free account

Why create an account?

  • Your WorldSupporter account gives you access to all functionalities of the platform
  • Once you are logged in, you can:
    • Save pages to your favorites
    • Give feedback or share contributions
    • participate in discussions
    • share your own contributions through the 7 WorldSupporter tools
Access level of this page
  • Public
  • WorldSupporters only
  • JoHo members
  • Private

JoHo kan jouw hulp goed gebruiken! Check hier de diverse bijbanen die aansluiten bij je studie, je competenties verbeteren, je cv versterken en je een bijdrage laten leveren aan een mooiere wereld