Achieving Outcomes Through Evidence Based (Informed) Practice: Iterative Intervention Logic (Programme Model) Development
(The I3Cycle)
Strategic Evaluation Working Paper
First Posted on 26 June 2004 as Version 1-2-3
Duignan, P. (2004, June 26). Achieving Outcomes Through Evidence Based (Informed) Practice: Iterative Intervention Logic (Programme Model) Development (The I3Cycle) [WWW document]. From The Strategic Evaluation Web Site. URL http://www.strategicevaluation.info/se/documents/123pdff.html
Dr Paul Duignan1
Parker Duignan Ltd
If using models from this paper in any context please source them to this document, the reference format is on the front cover.
Contents
2 Intervention Logic/Programme Modelling
3.1 The Planned Intervention Logic (Model)
3.2 The Research Evidence/Expert Opinion Informed Logic (Model)
3.3 The Implemented Intervention Logic
3.4 Next Iteration of the Planned Intervention Logic
Introduction
Throughout the world evidence based or evidence informed practice is being encouraged in all areas of public policy2. This paper is concerned with one aspect of facilitating evidence informed practice. It provides a framework (the I3Cycle) for thinking about the iterative cycle of intervention logic (programme model) development in programme planning, implementation and evaluation. It links intervention logic development with the use of information from research, empirical evidence and expert opinion.
In thinking about how evidence informed practice can be facilitated, the following factors need to be taken into account:
· A vast amount of evidence is available in the research literature on some topics, while on others, there is only limited evidence.
· Evidence exists in diverse forms across multiple research domains, individual programme evaluations, literature reviews and commentaries. For example, the available evidence on what causes positive educational outcomes for young people will focus on different types of programmes in different domains (e.g. education, health, poverty research, cultural research etc.).
· There are two types of evidence – research findings which are already available and those which are generated in the course of a particular instance of a programme being implemented.
· In those cases where evidence is absent or conflicting, recourse has to be had to expert opinion on what does and does not work.
· The development of evidence informed practice is an iterative (rather than one-off) process as evidence is progressively accumulated and various programme and policy approaches are tried in a range of settings.
The above picture presents some interesting challenges for undertaking evidence informed practice. How can the process of bringing evidence to bear on the development of a particular programme or policy be most usefullly conceptualised? This paper sets out the Iterative Intervention Logic Development Cycle (I3Cycle) as one way of conceptualising this process. The I3 Cycle can be used as a framework for developing evidence informed practice processes and also to assess evidence informed practice systems to see which part of the I3Cycle process they are focusing on or which parts they may be currently neglecting.
1 Intervention Logic/Programme Modelling
The I3Cycle assumes the use of an intervention logic or programme modelling approach to programme development. Intervention logic (programme modelling, programme theory, outcomes hierarchies, results chains) is now being promoted as good practice for a wide range of programmes, policies and activities. Intervention logic in its various guises sets out the intermediate steps or outcomes which need to be achieved in order to bring about a final outcome for a programme or policy. Diagram 1 provides a schematic view of an intervention logic formatted using the Backbone Outcomes Hierarchy approach3. In this type of format the logic sets out a cascading set of intermediate outcomes beneath higher level outcomes.
Intervention logic is a powerful tool which, if used properly, can provide a model of a programme; this can assist with stakeholder consultation, programme planning and implementation, outcomes measurement, indicator development, reporting, research and evaluation priority setting, and programme and policy evaluation.
The Iterative Intervention Logic Development Cycle (the I3Cycle) described in this paper is the iterative process of the sequential development of three types of intervention logic. These types are:
· Planned Intervention Logic – in what way is it planned that programme or policy will work.
· Research Evidence/Expert Opinion Logic – in what way does the research evidence and expert opinion say the programme or policy is most likely to work
· Instance-based Implemented Logic – how did (or did not) a programme or policy work in practice.
Information is brought to bear on the process of intervention logic/programme model development at various stages. Two major inputs are:
· Empirical evidence4
· Expert opinion.
Diagram 2 sets out the overall I3Cycle Model in which three types of intervention logic (programme models) are linked in an ongoing cycle of programme or policy improvement.
Each of the three types of intervention logic (programme model) is discussed below.
2.1 The Planned Intervention Logic (Model)
At the start of programme or policy development an initial intervention logic or programme model can be developed. This logic will be informed by the programme or policy planners’ knowledge of the sector in which they are working, their knowledge of the relevant research, expert opinion and their own views about what outcomes the programme is trying to achieve and how it is believed it will go about achieving these.
2.2 The Research Evidence/Expert Opinion Informed Logic (Model)
The second intervention logic (programme model) developed is one which results from subjecting the Planned Logic to assessment against the existing research literature and expert opinion (in those areas in which the research literature is not currently conclusive). This step in the cycle examines what evidence or opinion there is in regard to the steps set out in the intervention logic. It assesses how well the assumptions of the programme stand up against the research evidence which is available. In many areas there will not be definitive or clear research findings, in those cases expert opinion is drawn upon to assist in assessing the credibility of the steps set out in the Planned Logic.
2.3 The Implemented Intervention Logic
The third logic in the I3Cycle is the Implemented Logic (programme model). This is the logic which was actually implemented in the specific programme(s) being run. As with the Research/Expert Opinion Logic two types of information can contribute to this logic. The first is empirical evidence about changes in outcomes (either intermediate or final outcomes) which can be attributed5 to the programme. The second is expert opinion about what changes in outcomes could be attributed to the programme or policy. Expert opinion is needed in those many instances where it is not possible to empirically attribute changes to outcomes to the programme or policy.
2.4 Next Iteration of the Planned Intervention Logic
Following the development of the three types of intervention logic (programme models) set out above, the I3Cycle returns to the Planned Intervention logic (programme model). The findings from the early stages in the cycle are used to inform the development of a revised Planned Logic which then can be used as the basis for another iteration of programme or policy development.
Diagram 3 sets out a more detailed I3Cycle model. This model breaks out the separate intervention logics (programme models) which could be developed for a programme at the stage of the Research Evidence/Expert Opinion Logic and the Implemented Logic. These consist of a research/empirical evidence model and an expert opinion model at each stage.
This paper has described the I3Cycle for Evidence Informed Iterative Intervention Logic (Programme Model) Development. This model can be used for planning and implementing evidence informed processes. Particular instances of evidence informed practice within Managing for Outcomes will be more or less complete in regard to the elements of the I3Cycle included. The I3Cycle can be used to assess which elements of the cycle are being used in a particular instance of evidence informed practice; this then allows recommendations to be made for the process to be enhanced to include all the required elements set out in the I3Cycle.
4 References
1 Dr Paul Duignan works for Parker Duignan Ltd Wellington and is a Senior Research Fellow at the Centre for Social and Health Outcomes Massey University Auckland, New Zealand (SHORE).
2 While much of the emphasis on evidence based practice has focused on the public and NGO sector, the principles in this paper apply equally to parallel instances in the private sector.
3 There are numerous textual, tabular and diagrammatic approaches to setting out intervention logics for a programme or policy. All are intended to provide a model of the way in which it is believe the programme will work to achieve its final outcomes.
4 Empirical evidence in this context is not restricted to a narrow range of quantitative or experimental methodologies. However while recognising, in philosophy of science terms, that observation is always to some extent theory laden, it refers to information gained from observation of what happens in practice rather than purely theoretical discussion of what might or should be.
5 The issue of attribution of changes in outcome indicators to the activity of a specific programme or policy is often neglected and the mere collection of outcome indicators from a programme is believed to say something definitive about the efficacy of a programme or policy. For further discussion of the question of attribution see Duignan (2004).