To P, or not to P, Ravenzwaaij (2016)

The Null Hypothesis Significance Test (NHST) is used to show the probability of an event given that the null hypothesis is true. The P-value communicates this probability. In this article, 4 problems are discussed regarding the use of P-values to communicate probabilities.

Why should we question the use of P-values?

For example, when effects of medication are compared to a placebo the group difference has a P-value of 0,12. People in the medicated group report less symptoms. The P-value in this example means that when the medication is just as effective as the placebo, the chance of finding a difference is 12%. This number is too low and means that the null-hypothesis cannot be rejected. A low value for P is typically below 0.05 and only then the finding is statistically significant. However, p-values are associated with several problems.

Problem 1:

The use of P-values fails to quantify proof in favour of the null-hypothesis because there are two explanations of a non-significant value of P. The first is the lack of statistical power, the second is that the null-hypothesis is true. For example, in medical experiments evidence in favour of the null-hypothesis should be considered. When the same experiment is conducted in twenty different environments, with the ‘lucky’ one concluding there is an effect while there is not, this could potentially be harmful for future patients. 

Problem 2:

Outcomes of P-values cause over-rejection of the null-hypothesis. The problem here is that evidence is a relative concept. When you only consider data that lies under the null-hypothesis this will lead to biases in decision making. One example is the case of Sally Clark. Her two sons died in infancy in two separate incidents. The argument for convicting her for murder was the statistical evidence the probability of two infants dying from sudden infant death syndrome is extremely low. However, the chance of a mother murdering her two infant children is even less. The trust in one piece of evidence can cause neglection of even stronger evidence.

Problem 3:

The p-value only gives an abstract value of data. If cannot infer something about the hypothesis. The only thing the p-value can be used for is to see the probability of a data-pattern that is as extreme as the observed data pattern based on that sample size. Therefore, p-values are often incorrectly interpreted.

Problem 4:

The fourth problem is that the use of p-values does not allow for optional stopping. The p-value can only be obtained after the sample size is determined and the statistical analysis is carried out. However, in practice additional participants are often tested when the p-value approaches significance. If researches retest repeatedly, they are guaranteed to find a statistically significant result whether the null-hypothesis is true or not.

One potential solution is the use of the Bayes factor. The Bayesian way of hypothesis testing uses two competing hypothesis and the relative comparison is used as Bayes factor. This provides an alternative to each of the four problems listed above.

Summary: To P, or not to P, Ravenzwaaij (2016)

• The Null Hypothesis Significance Test (NHST) is used to show the probability of an event given that the null hypothesis is true. The P-value communicates this probability.
• The use of P-values fails to quantify proof in favour of the null-hypothesis because there are two explanations of a non-significant value of P. The first is the lack of statistical power, the second is that the null-hypothesis is true.
• The p-value only gives an abstract value of data. If cannot infer something about the hypothesis. The only thing the p-value can be used for is to see the probability of a data-pattern that is as extreme as the observed data pattern based on that sample size. Therefore, p-values are often incorrectly interpreted.

Study note: To P, or not to P, Ravenzwaaij (2016)

• In the article there are four problems described. Make sure to be able to name and describe the four issues with p-values.
• There is a solution mentioned that solves all four of these problems, what is it?