Evidence-based Clinical Practice – Lecture 2 (UNIVERSITY OF AMSTERDAM)

In the case of a large number of tests, the probability of a false positive does not change for individual tests. However, the tests taken together have an inflated probability of false positives (i.e. inflated type-I error rate). This is also called the multiple comparison problem.

A research needs to have enough power. Besides that, statistical significance does not tell us anything about practical significance as any result can be significant when the sample size is sufficiently large. A large sample size will almost always lead to significant results and this does not tell us anything about the practical relevance. This is partially due to the arbitrary cut-off point for the p-value (i.e. p<.05). The cut-off point is not adjusted for the sample size. Therefore, the p-value is not indicative of practical relevance.

Practical significance can be assessed by considering factors such as clinical benefit (1), cost (2) and side-effects (3). This requires the effect size and the risk potency.

The absence of evidence does not imply evidence for absence. This means that not finding evidence does not mean that there is no evidence. It is possible that the study is underpowered. Furthermore, it is possible that there are small, meaningless differences which are significant and large, meaningful differences which are not significant.

The power refers to the probability of finding an effect when there actually is an effect (equal to the size of power). This is mostly considered when failing to reject the null hypothesis but it should also be considered when the null hypothesis is rejected.

The power affects the capacity to interpret the p-value. The p-value exhibits a wide sample-to-sample variability unless the statistical power is very high. This implies that evaluating a study by the p-value alone is spurious. A research with power of 50% (i.e. standard in psychology) leads to a less than 50% odds of replicating the significant result in the study. This contributes to the replication crisis. In short, the p-value does not provide information regarding the strength of evidence against the null hypothesis.

The p-values are typically only meaningful with large samples. Therefore, it is useful to look at the effect size. The effect size refers to how strong the effect of an intervention is. It corresponds to the degree of non-overlap between sample distributions (1) and the probability that one could guess which group a person came from, based only on their test score.

Typical methods of denoting the effect size are Cohen’s d (1), Hedges’ g (2) and Pearson’s r (3).

An intervention that is compared to a placebo requires an effect size of 0.8. However, an intervention compared to another intervention requires an effect size of 0.5. Effect size is heavily inflated in small samples and thus requires large samples.

The less overlap there is in a graph, the larger the effect size. However, the effect size of an intervention does not tell us how many people recovered after an intervention.

The effect sizes for discrete outcomes (e.g. recovered or not) should be interpreted within clinical norms for health. Effect sizes for discrete outcomes make use of the odds ratio (1), number needed to treat (2) and area under curve (i.e. AUC) (3). These effect sizes are more pertinent to clinical significance. The disadvantage of effect sizes of discrete outcomes is that they are very sensitive to the limits that are set by the researchers (e.g. cut-off point).

The disadvantages of the r and the d effect sizes are that they are relatively abstract (1), they are not intended as measures of clinical significance (2) and they are not readily interpretable in terms of how much the individuals are affected by treatment (3).

The dichotomization of continuous data leads to a loss of information (1), arbitrary effect size indexes (2) and inconsistent effect size indexes (3). This is mostly due to the cut-off point of failure (e.g. treatment is not effective).

There are several different types of other effect sizes:

1. Odds ratio
   This refers to the odds of falling within a category for group A or group B (e.g. the odds of falling in the recovered category is greater for the intervention group than for the placebo group). A disadvantage is that the magnitude may approach infinity if the outcome is very rare, very common or near random. Furthermore, the magnitude varies strongly with the choice of cut-off point.
2. Risk ratio
   This effect size is obtained by dividing the failure or success rate of the comparison group by the failure or success rate of the treatment group. The choice of cut-off (i.e. success or not) influences the magnitude of the risk ratio.
3. Relative risk
   This refers to the odds of improvement in the control condition divided by the odds of improvement in the intervention condition. This ratio should be 1 or lower.
4. Relative risk reduction
   This refers to a reduction in the incidence of negative outcomes. It is 1 minus the relative risk. This should be as close to 1 as possible.
5. Risk difference (i.e. absolute risk reduction)
   This refers to the percentage of failures in the treatment group minus the percentage of failures in the comparison group. However, it can also use successes rather than failures and the risk difference is often near zero when the odds ratio and the risk ratio are very large.
6. Number needed to treat (NTT)
   This refers to the number of people who need to be treated in order to generate one more success or one less failure than would have resulted if everyone received the comparison treatment. In risk studies, it is the number who would have been exposed to the risk factor to generate one more case than if none had been exposed. Number needed to treat can only be interpreted relative to the comparison.
7. Area under curve (AUC)
   This refers to the probability that a randomly selected subject in the treatment group has a better outcome than one in the comparison group. This can be computed using clinical judgement alone.

Risk refers to the probability that the intervention group does worse than the control group. Individual improvement refers to how many individuals improved or deteriorated.

The meta-analysis gives a summary of the literature. It assesses which variables explain the differences between different research papers. It is important for clinical practice as there is often no time to read all research papers. There are two choices when summarizing the literature:

1. Narrative review
   This is reading the papers and giving a short summary of all the articles. However, the disadvantages are that there is a focus on the p-values in the original studies (1), it does not necessarily cover inconsistencies over studies (2), it does not deal with differences in reliability (3) and it is often only based on published literature (4). It is tempting to write that that studies support your theory even if they do not fully do.
2. Meta-regression
   This is giving a more overall overview of the literature. It makes use of statistical analysis (e.g. GMA). In a meta-regression, the single p-values are irrelevant (1), the overall effect is reduced by inconsistent outcomes (2), the studies are weighted by reliability (3) and there is extensive literature research and corrections for the file drawer problem (4).

Meta-regression can make use of a regression analysis. In the case of a regression analysis, the intercept (i.e. b0) is the value that is expected if there is a score of 0 on all independent variables. In case that there are no independent variables in the regression analysis, the model looks as follows:

In this case, y is the effect size, b0 is the overall effect and e(i) is the error. The model looks as following if an independent variable is added:

In this case, b1 (i.e. slope) tells us something about the effect of the independent variable on the intercept (e.g. overall effect size decreases with decreasing independent variable).

There are several steps in a meta-regression:

1. Ask the right question
2. Find studies (including in the file drawer)
3. Determine inclusion criteria (e.g. only English papers)
4. Choose dependent variable from studies (e.g. follow-up anxiety scores)
5. Choose effect size in meta-regression (i.e. for continuous or discrete outcome)
6. Choose independent variable in meta-regression (e.g. mean age)
7. Do the meta-regression
8. Report results

There are three methods to do the meta-regression:

1. Regression analysis
   This can be done in SPSS. The reliability of the studies is not taken into account but the variance between studies is taken into account.
2. Meta-regression: fixed effects
   This can be done in SPSS using a script. The reliability of the studies can be taken into account but the variance between studies are not taken into account.
3. Meta-regression: random effects
   This can be done in SPSS using a script. The reliability and the variance between studies can be taken into account.

If the reliability of the studies is not taken into account in a meta-regression, then each study is treated as if they are equally reliable. This leads to unreliable studies having an unduly influence. If the variance between studies is not taken into account in a meta-regression, then the result will often become significant. This is because every study is treated as if random variation between studies does not exist.

The random-effects meta-regression is the best option for a meta-analysis.  However, it is important to note that meta-regression is not an experiment and causal conclusions cannot be drawn. Meta-regression may highlight moderators of intervention success which have not been investigated directly.