Which factors are important in evaluating treatments? - Chapter 4

What is this chapter about?

What works for whom, and why? In particular, a lot of research has been conducted into the question: what is the impact of a particular treatment compared to a) no treatment, and b) other types of treatments?

What makes a treatment effective?

It is difficult to pinpoint the factors that make treatments effective, due to the following factors:

• When a child's mental health improves after treatment (correlation), it does not mean that the treatment is the cause ("correlation ≠ causation").
• After all, it is also possible that the improvement is due to a third (confound) variable, through psychosocial development and/or by spontaneous recovery.
• Regression to the mean: it is very likely that someone who initially reports many problems will automatically mention fewer problems later on. This also applies to questionnaires taken shortly after each other.
• Selection bias of individuals who are treated or not treated, as opposed to those who stop treatment halfway through, can lead to an incorrect and misleading high proportions of "effective" treatments.
• Specific and nonspecific effects of treatments are sometimes difficult to distinguish.

Currently, two approaches are used to assess the outcomes of a treatment: RCTs, and to analyse routinely collected results (using data from databases).

What are RCTs and how is data analysed from big databases?

RCTs are seen as the most powerful source of evidence ("the gold standard"), because the researchers here also take into account other influential factors besides treatment. This allows them to assign the differences found to the treatment and therefore be able to equate treatment with causation, thus resolving the first factor. By randomly dividing the subjects into groups, any unknown third variables are distributed fairly. A common criticism about RCTs is that the people who participate in such studies are not necessarily representative of clinical practice: they often have less serious problems than participants that really suffer from mental disorders.

An additional method to evaluate treatments is through the use of routinely collected results from large databases. Symptom scores obtained before and after treatment are compared. If someone would use this method only, the absence of random assignment of participants is a big problem. This problem can be solved by using data from RCTs or from studies that use a naturalistic control group.

A major advantage of analysing outcome data compared to an RCT design is that it also gives the possibility of analysing processes that cannot be easily and ethically manipulated or randomly classified, such as engagement and therapeutic alliance.

How can we decide if an outcome is clinically significant?

Whether the data comes from RCT or from naturalistic studies, it is always difficult to determine what is sufficient as a meaningful (clinically significant) outcome. After all, a statistically significant outcome is not always clinically significant. Some researchers have developed indices to address this. The two most commonly used methods are:

• Look at recovery: This means that symptom scores from before treatment were (too) high, and the scores are lower/average after treatment.
• Assessing reliable change: analysing the amount of change that can be attributed to the treatment.

Which criticism is there on the evaluation of treatments?

Although, on the one hand, there is a lot of interest in making the routine evaluation of a treatment a standard part of clinical practice, there is also a lot of criticism. Many therapists do not want to use standardized questionnaires to evaluate the impact of treatment. There are several reasons for this, for example 1) because they feel that this method of evaluation cannot capture complexity, 2) because this is an administrative burden which limits the time for personal contact with the client, 3) because the data can be used for purposes that are not in the client's interest (where any reported improvement can lead to dismissal of care and refusal of (further) services).

What about outcomes?

Previously, routine evaluations of treatments were obtained from one or more questionnaires completed by the therapist. However, this is not very helpful, since therapists have their own styles. Therefore, the focus is now more on PROMS (patient-reported outcomes) or PREMS (patient-reported experiences). Teacher reports can also be useful, as teachers can accurately observe externalizing symptoms (such as aggression). However, they cannot properly assess the internalizing symptoms (such as anxiety). Parent reports can also be useful. Although there is a bias (because they are also unable to properly assess the internalising problems and/or assess their child from their point of view), it can be helpful when the children are too young to complete reports on their own.

It is recommended that therapists collect data from multiple persons (at least from the child him/herself, the parent/caregiver and the therapist) with the greatest value attached to the reporting of the child. What is annoying is that in general (in less than 75% of cases) there is no agreement between the perspectives of the child, the parent or the practitioner in terms of the problems for which help was initially sought.

When determining the outcome, it is necessary to carefully determine the domains that are evaluated (e.g. symptom reduction, improved functioning at school or at home), since improvement in one domain does not guarantee improvement in another domain.

The first measurement of a domain takes place when the child first enters the therapy. The second measurement takes place at a time that is predetermined, for example at the end of treatment, or six months after treatment. The problem here is the low response rate of people at the second time of measurement: the data that is missing from the second measurement is not random and therefore implies a systematic difference between two groups. According to Clark and colleagues (2008), people who do not submit data to the second measurement have often not been involved (enough) in the treatment or are somehow dissatisfied with the treatment, which will result in conclusions of these unfair dates resulting in an overestimation of the "effectiveness" of the treatment.

How is information obtained from outcome analyses used for practice?

What can we do for practice, with the information obtained from outcome analyses? In principle, there are three possibilities:

1. For example, if therapists looked at the data qualitatively, they could find out which techniques are generally related to a better outcome (for example, when patients feel that they are being listened to).
2. Follow an individual's trajectory: during each session, the progress is mapped and feedback is given about the session. If there is no change after, say, five meetings, according to Miller and colleagues (2006) it makes sense for another therapist to be deployed to prevent a bad outcome and drop-out.
3. Use a bundle of outcome data for benchmarking: results can also be compared (e.g. annually) between institutions.

How are academics and practitioners brought together?

According to the authors of the book, there are three promising ways to better analyse therapies for children and young adults:

1. Analysing individuality in terms of change trajectories. When evaluating psychotherapies, the average changes of groups are often looked at. As a result, actual effects on certain subgroups may be over- or underestimated. In addition, there is also often a lot of variation within (sub)groups in terms of, among other things, reported improvement. Thus, instead of a top-down approach (group), a bottom-up approach (individual) might be a better way to determine the actual effects of a therapy.
2. Analysing more nuanced models of processes and outcomes. Due to the enormous focus on (wanting to develop) causal models for specific diagnoses, there is little attention on more nuanced models about change processes and outcomes. In such a nuanced model, the focus is on networks of symptoms that are determined by multiple factors and also influence multiple factors. By mapping such networks (links and interactions with symptoms) – at an individual or group level – more insight can be obtained into the areas in which change is possible. An example of this story is the development of personalised methodology: although pen-and-paper questionnaires are still the norm, there is increasing attention to online questionnaires. A major advantage of online questionnaires is that the questions can be extracted from an item bank, and that the selected question depends on a previously given answer.
3. The further development of the links between academics and practitioners. Practitioners should be more aware of the results found by academics, and vice versa.

What works for whom, and why? In particular, a lot of research has been conducted into the question: what is the impact of a particular treatment compared to a) no treatment, and b) other types of treatments?