Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience – Button et al. - 2013 - Article

What is the purpose of this paper?

A study with low power has less likelihood of detecting a true effect, but it is less well appreciated that low power also decreases the likelihood that a significant result reflects a true effect.  This paper shows that average statistical power of studies in neurosciences is low. Consequences include: overestimates of ES and low reproducibility. There are also ethical dimensions: unreliable research and inefficient/wasteful.

The paper discusses issues that arise when low-powered research is frequent. They are divided into two categories: (1) concerning problems that are mathematically expected even if the research is otherwise perfect, (2) concerning problems that reflect biases tending to co-occur with studies of low power or become worse in small, underpowered studies.

What are the three main problems?

The main problems contributing to producing unreliable findings with low power are:

• Low probability of finding true effects
• Low positive predictive value (PPV) when an effect is claimed
• An exaggerated estimate of the effect size when a true effect is discovered

What are some key statistical terms?

• CAMARADES: collaborative approach to meta-analysis and review of animal data from experimental studies. A collaboration aiming to decrease bias and improve method quality and reporting in animal research. Promotes data-sharing.
• Effect size: standardized measure quantifying the size of a difference between two groups or strength of an association between two variables.
• Excess significance: phenomenon where literature has an excess of statistically significant results due to biases in reporting. Mechanisms contributing to bias: study publication bias, selective outcome reporting, and selective analysis bias.
• Fixed and random effects: a fixed-effect meta-analysis assumes that the underlying effect is the same in all studies and that any variation is because of sampling errors.
• Meta-analysis: statistical methods for contrasting and combining results from different studies to give more powerful estimates of the true ES.
• Positive predictive value (PPV): probability that a ‘positive’ finding reflects a true positive effect; depends on prior probability of it being true.
• Proteus phenomenon: the situation where the first published study is often most biased towards an extreme result (winner’s curse). Replication studies tend to be less biased toward the extreme – find smaller ES’s of contradicting effects.
• Statistical power: probability that a test will correctly reject a false H0. The probability of not making a type II error (probability making type II error = β and power = 1 – β).
• Winner’s curse: the ‘lucky’ scientists who makes a discovery is cursed by finding an inflated estimate of the effect. Most severe when the thresholds (significance) are strict and studies too small – low power.

What are additional biases associated with low power?

Firstly, low-powered studies are more likely to give a wide range of estimates of the magnitude of an effect. This is known as ‘vibration of effects’: situation where a study obtains different estimates of an effect size depending on the analytical options it implements. Results can vary depending on the analysis strategy, especially in small studies.

Secondly, publication bias, selective data analysis, and selective reporting of outcomes are likely to affect low-powered studies.

Third, small studies may be lower quality in other aspects of their design as well. These factors can further exacerbate the low reliability of evidence attained in studies with low power.

What are the implications for the likelihood that a research finding reflects a true effect?

Trying to establish the average power in neuroscience is hampered by the issue that the ES is unknown. One solution may be using data from meta-analysis (estimate ES). Studies contributing to meta-analysis, however, are subject to the same problem of unknown ES.

Results show that the average power of studies in neuroscience is probably not more than between 8%-31%. This has major implications for the field.

• Likelihood that any significant finding is actually a true effect is small (PPV decreases when power decreases).
• Ethical implications: inefficient and wasteful in animal/human testing.

What recommendations are there for researchers?

• Perform an a priori power calculation: estimate ES you are looking for and design your study accordingly.
• Disclose methods and findings transparently.
• Pre-register your study and analysis plan: this clarifies confirmatory and exploratory analyses, reducing opportunities for non-transparency.
• Make study materials and data available – improving quality of replication.
• Work collaboratively to increase power and replicate findings; combining data increases total N (and power) while minimizing labour/resources.

What can we conclude?

Researchers can improve confidence in published reports by noting in the test how they determined their sample size, all data exclusions, all data manipulations, and all measures in the study. When stating that is not possible, disclosure of rationale and justification of deviations will improve readers’ understanding and interpretation of reported effects and what level of confidence in the reported effects is appropriate.