The success phase: How to evaluate the intervention? - Chapter 6

To assess whether the problem that was targeted has indeed changed for the good, the last step of the PATHS model involves the evaluation of the intervention. Most importantly, it must be examined whether the intervention had the expected effects in terms of the outcome variable. A solid evaluation of the intervention is important for several reasons.

• If the intervention was successful, the practitioner may in the future repeat the intervention if the problem re-occurs.
• If the evaluation demonstrates that the intervention is effective at relieving or solving a problem, this usually helps to persuade potential investors to subsidize future implementations of the intervention.
• Additionally, as technological possibilities evolve, different channels and strategies for the distribution of the intervention may be needed to optimally expose members of the new cohort to the intervention.

Especially when the intervention is likely to be repeated in the future, it is wise to communicate the outcomes of the evaluation to all those involved in the development and distribution of the intervention. The evaluation helps the practitioner also pinpoint what went wrong, whether it’s worthwhile investing time, money, and energy in adjusting the intervention for future use, and what action must be undertaken to adjust the intervention and/or distribution.

What different types of evaluation exist?

At least two types of evaluation are important.

Effect Evaluation

Here we assess the extent to which variables that are directly related to the problem have changed over time. At least the effect of the intervention on the specified outcome variable in the process model should be assessed. Additionally, to assessing the outcome variables and variables related to it, it may also be wise to assess those variables from the process model that, in the Help phase, were selected as targets for the intervention. This helps to better understand the effect of the intervention on the outcome variable.

There are four different potential scenarios we want to describe here.

• The most ideal scenario is that the intervention both affects the selected causal variables from the process model and the outcome variable in the desired direction. In this case, the change in the outcome variable most likely can, at least partially, be attributed to the change in the causal variables selected from the process model. Then the practitioner can conclude that the line of reasoning behind the process model and the balance table seems to be correct, and the intervention was sufficiently effective.
• The intervention affects the selected causal variables from the process model in the desired direction, but it doesn’t influence the outcome variable. This would mean that, from the perspective of the outcome variable, the intervention has failed. Nonetheless, the evaluation shows some degree of success: the selected causal variables from the process model were effectively targeted and show a positive change. perhaps the practitioner chose the wrong outcome variable or didn’t find or select the most important causes of the outcome variable and therefore must scrutinize the choices made in the P, A, or T stages of the PATHS model and adjust them.
• The intervention affects the outcome variable in the desired direction but has no impact on the causal variables from the process model that were the focus of the intervention. The intervention is a success from the perspective of the outcome variable, but it did not work in the way it was intended.
• The most negative scenario is that the intervention doesn’t produce any change in the desired direction or in the outcome or in the selected causal variables from the process model. In this case, many things have probably gone wrong. The line of reasoning behind the process model and the balance table may not be correct, or the intervention may not have been implemented in a way that was sufficient for any effects to occur. Before concluding that an intervention has failed, the practitioner must take a good look at the way the outcome variables and other variables were assessed.

Ideally, the evaluation of the effectiveness of an intervention will be conducted with a study using a carefully controlled research design and procedure. The evaluation takes place on a much larger scale compared to the pilot study, unfolds during a longer time, and encompasses many more members of the target group. However, it is not always possible to randomly assign members of the target group to an experimental and control condition in field settings. The lack of random assignment has consequences for the strength of the evidence that can be found for the intervention’s effectiveness. Without random assignment it’s possible that structural differences between groups cloud or interact with the effect of the intervention. Despite this limitation, evaluation research following a quasi-experimental design may provide relatively strong evidence for an intervention’s effectiveness, although somewhat less strong than a full experimental design.

For whatever reason, sometimes only an evaluation in terms of a post-test design is feasible. That is, following the implementation of the intervention, the outcome and related variables are assessed among members of the target group. Based on a post-test design a practitioner cannot make any solid claims about the effect of the intervention on the outcome variable or related variables. A post-test may only help evaluate the intervention in terms of perceived effectiveness.

Process Evaluation

The process evaluation is sued to monitor and evaluate the implementation of the intervention. It is important in understanding why an intervention did or did not work. Especially if the effect evaluation shows that the outcome variable didn’t change as intended, the process evaluation may help understand why this is the case. For example, maybe certain distributors didn’t do their job well and that, consequently, only a small part of the target group was reached. Process evaluation happens both during and following the implementation of the intervention. It is recommended that in the context of process evaluation, at least the following topics should be assessed or evaluated.

• Completeness of the implementation: information on the degree to which all practical applications of the intervention were implemented as intended.
• Exposure to intervention: information on the degree to which members of the target group actively engaged in, interacted with, were receptive to, and/or used materials or recommended resources.
• Satisfaction with the intervention: degree to which members of the target group positively evaluated the intervention in terms of, for instance, user friendliness or attractiveness.
• Recruitment: information on the procedures that were used to attract members of the target group to the intervention and to get them involved.
• Context: information on environmental factors that may have influenced the implementation of the intervention or affected the intervention’s outcomes.

Why is evaluation planning, implementation, and reporting important?

What type of evaluation is feasible will depend on resources, time, and money. Whatever choices are made concerning the evaluation, it’s advisable to thoroughly think through and plan the evaluation before the intervention is implemented. A timely planning of the evaluation is relevant since evaluating is not an activity that merely takes place following the intervention.

Concerning the effect evaluations, members of the target group must be recruited for assessment. An evaluation report informs all those involved in the design and implementation process about the outcomes of the intervention. It allows those who contributed to the design and implementation process to see what their contribution led up to. For financial investors, the evaluation report justifies the time and money invested in the intervention. Additionally, a successful implementation may also be communicated by means of a press release or coverage in various media channels.