Kraemer et al. (2003). Measures of clinical significance.” - Article summary

There are three important questions when assessing the relationship between variables:

1. Statistical significance
2. Effect size
3. Practical significance

The practical significance (i.e. clinical significance) can be assessed by considering factors such as clinical benefit (1), cost (2) and side effects (3). In order to assess the practical significance, the strength of the association (i.e. ‘r’) (1), magnitude of the difference between the difference between treatment and comparison (i.e. ‘d’) (2) and measures of risk potency (3) can be used.  Risk potency can be assessed using the odds ratio (1), risk ratio (2), relative risk reduction (3), risk difference (4) and number needed to treat (5).

The p-value refers to the probability that the found outcome or more extreme is found, given that the null hypothesis is true. It is possible to have statistical significance by chance and outcomes with lower p-values are sometimes interpreted as having stronger effect sizes. A non-significant result also does not tell us anything about the truth of the null hypothesis.

There are several different effect size measures:

1. The ‘r’ family
   This is expressing effect size in terms of strength of association (e.g. correlation).
2. The ‘d’ family
   This is an effect size that can be used when the independent variable is binary and the dependent variable is ordered. It can be computed by subtracting the mean of the second group from the mean of the first group and divide it by the pooled standard deviation of both groups. The effect size ranges from minus to plus infinity.
3. Measures of risk potency
   This is an effect size that can be used when both the dependent and independent variables are binary. The odds ratio and risk ratio vary from 0 to infinity and 1 indicates no effect. Risk relative reduction and risk difference vary between -1 and 1 with 0 indicating no effect. Number needed to treat (NNT) ranges from 1 to plus infinity with very large values indicating no treatment effect.
4. AUC
   This is an effect size that can be used when the independent variable is binary but the dependent variable is either binary or ordered. It ranges from 0% to 100% with 50% indicating no effect.

This standard is relative and it should be noted that ‘larger than typical’ should be used rather than ‘large’. However, it might be best to find information about typical effect sizes in the context of a particular research field.

The disadvantages to the ‘d’ and the ‘r’ as measures of clinical significance are that they are relatively abstract (1), they were not intended as measures of clinical significance (2) and they are not readily interpretable in terms of how much individuals are affected by treatment (3).

There is no consensus on the externally provided standards for the clinical significance of treatments. Clinical significance could be defined as a change to normal functioning due Phi is not a good measure for clinical significance.

When continuous data are dichotomized (e.g. success or failure), there is a loss of information (1), it can result in arbitrary effect size indexes (2) and it can result in inconsistent effect size indexes (3), mostly due to different choices for the cut-off point of failure.

A limitation of the odds ratio is that the magnitude of the odds ratio may approach infinity if the outcome is very rare or very common. This can also happen if the outcome is near random. The magnitude of the cut-off point varies strongly with the choice of cut-point. Therefore, the interpretation of the odds-ratio is arbitrary and very difficult.

The risk ratio is obtained by dividing the failure or success rate of the comparison group by the failure or success rate of the treatment group.  The choice of cut-off point changes the magnitude of the risk ratio which makes it difficult to interpret.

Relative risk reduction is computed by subtracting the treatment group failure rate from the comparison group failure rate and dividing it by the comparison group failure rate or by using the success rates. There are no agreed upon standards for assessing the magnitude of this.

Risk difference is computed by subtracting the percentage of failures in the treatment group from the percentage of failures in the comparison group or by using the successes. The risk difference is often near zero when the odds ratio and the risk ratios are very large.

Number needed to treat refers to the number of patients who must be treated to generate one more success or one less failure than would have resulted if all people had been given the comparison treatment. In risk studies, NNT is the number who would need to be exposed to the risk factor to generate one more case than if none had been exposed. NNT can only be interpreted relative to the comparison.

AUC represents the probability that a randomly selected subject in the treatment group has a better result than one in the comparison group. AUC can be computed by using clinical judgement alone.