Systematic Outcomes Analysis (in the Evaluation (outcome) building-block of the system) uses an exhaustive set of seven possible outcome evaluation designs. This list can be used to establish exactly what outcome evaluation is, and what outcome evaluation is not possible, for any intervention. High level outcome evaluation questions are identified in Systematic Outcomes Analysis and are examined to see if any of the seven possible outcome evaluation designs are appropriate, feasible and affordable.
The seven possible outcome evaluation designs are:
Design 1: True experiment design.
Applying an intervention to a group (intervention group) and comparing it to a group (control group) which has not received the intervention where there is no reason to believe that there are any relevant differences between the groups (e.g. through random assignment to to intervention and control group).
Design 2: Regression discontinuity design.
Applying an intervention only to a group (intervention group) who are the 'worst off' in terms of their initial levels on some outcome of interest. The results for this group are then compared to the results for a wider untreated group (control group) and they should have improved relative to the control group.
Design 3: Time series analysis design.
Tracking an outcome of interest over many observations in a situation where the intervention starts at a clearly specified point in time. If a clear change in the series of observations is observed at the time when the intervention starts, this is regarded as evidence that the intervention had an effect.
Design 4: Constructed comparison group design.
Identifying a 'group' which is similar in as many regards as possible to the group receiving the intervention. This can include either identifying other actual groups, or constructing a nominal control 'group' of what would have happened to those receiving the intervention if they had not, in fact, received it (e.g. propensity matching).
Design 5: Exhaustive causal identification and elimination design.
Systematically and exhaustively looking for all the possibilities which could have caused a change in outcomes and eliminating these alternative explanations in favor of the intervention as the best explanation for what happened. This approach needs to go well beyond just developing an explanation as to why the intervention could have worked without dismissing all alternative explanations which can be identified. Sometimes called a 'forensic' evaluation method.
Design 6: Expert judgement design.
Asking experts to judge whether they think that the intervention caused the outcomes by using whatever way they believe is appropriate to make this judgement. (This design, along with some of the other outcome designs in some instances, is rejected by some stakeholders as not being a valid way of determining whether an intervention actually caused high level outcomes. It is included in the list of outcome evaluation designs here because it is accepted by other stakeholders as actually doing this.)
Design 7: Key informant judgement design.
Asking key informants (a selection of those who are likely to know what has happened) to judge whether they think that the intervention caused the outcomes and allowing them to do this in whatever way they believe is appropriate to make this judgement. (This design, along with some of the other designs in some instances, is rejected by some stakeholders as not a valid way of determining whether an intervention actually caused high level outcomes. It is included in the list of evaluation designs here because it is accepted by other stakeholders as actually doing this.)
Of these designs, the first four can be used to estimate effect sizes. Effect sizes are a quantitative measurement of the amount an intervention has changed an outcome. Estimated effect sizes are essential for carrying out some of the elements in other building blocks. The Prerequisites building blocks 5-8 diagram sets out the prerequisites here.
[Note: This list of designs is still provisional within Systematic Outcomes Analysis. The first five were derived from the work of Michael Scriven identifying causal evaluation designs. The last two have been added because they are accepted by some stakeholders in some real-world circumstances as providing evidence that an intervention has caused high level outcomes. Different disciplines use different terms for these types of designs and therefore the names of these designs within Systematic Outcomes Analysis may be changed in the future. For instance, general regression analyses as often undertaken in economic analysis are currently included under the 'constructed comparison group design' this may or may not be a good idea, it may be that they should have their own design type in this list. Comment on whether this is actually an exhaustive list of designs would be appreciated (send to paul (at) parkerduignan.com).]
[V1.1.2]