This is a technical paper which outlines general principles in regard to linking research and evaluation planning and results dissemination to a government organisation’s Statement of Intent (SOI) and the use of a methodology for doing this – the Research Evaluation and Monitoring Intervention Logic Outcomes methodology (REMLogic Outcomes methodology). Such linking is important not only for research and evaluation planning but also for ensuring that the results from research and evaluation are disseminated and implemented. The paper discusses various technical issues related to the type of intervention logic which is being developed under a Statement of Intent. In particular it looks at the question of measurement and attribution in the development of outcome hierarchies which can be used with REMLogic. It identifies a set of general principles in linking research and evaluation to an SOI. These are: being clear about the nature of the SOI and its underpinning intervention logic; facilitating a quality stakeholder research and evaluation priorities discussion; developing a knowledge management infrastructure; undertaking research and evaluation capacity building to enable identification of priorities; and allowing for three levels of measurement and evaluation within a SOI.

1 Advantages of linking research and evaluation (R&E) to the Statement of Intent (SOI)

There are a number of advantages in linking research and evaluation planning, dissemination and implementation to an organisation’s SOI, these are:

· The SOI sets out the high-level outcomes for the organisation and they (together with the organisation’s intervention logics – the rationale for the underpinning strategies and activities) should be what drive the overall direction of the organisation’s research and evaluation effort.

· In the NZ State Sector, the SOI and its underpinning intervention logic(s) is likely to become central to the way in which an organisation is conceptualised by both internal and external stakeholders. For research and evaluation planning, dissemination and implementation to be relevant to the organisation it needs to be linked to the way in which stakeholders think about/conceptualise an organisation.

· A learning organisation needs to have feedback mechanisms to ensure that research and evaluation results lead to improved organisational practice. A useful way of doing this is to link research and evaluation activities to the intervention logics which should lie beneath the SOI. When research and evaluation findings become available such linking ensures that stakeholders immediately know the specific area of the organisation’s function to which the R&E findings should be applied and they are therefore more likely to implement them as a part of ongoing strategic planning.

2 Actually Linking R&E with an SOI

2.1 How R&E can be linked to the SOI

The concept of linking research and evaluation with the Statement of Intent is simple in theory but sometimes more difficult in practice. Ideally a Statement of Intent will include high-level outcomes for an organisation linked to high-level outcomes for the sector in which it operates and above that higher-level government societal outcomes. Beneath this structure will be an intervention logic(s) which cascades down through intermediate outcomes setting out the links between such intermediate outcomes and higher-level outcomes. The Research Evaluation and Monitoring Intervention Logic Outcomes Methodology (REMLogic Outcomes methodology) uses an outcomes hierarchy-based intervention logic to identify: 1) strategic indicators (not necessarily attributable); 2) attributable performance indicators; and 3) research and evaluation priorities linked specifically to parts of the outcome hierarchy. A simplified logic structure is set out in Diagram 1.² As can be seen from this diagram, it is possible to use it as a framework for firstly, outcome measures (sometimes called indicators) and secondly for visualising where priority research projects link to the intervention logic.

Diagram 1: REMLogic Outcomes methodology for linking intervention logic, outcome measures/indicators and priority research and evaluation projects

2.2 Need for a sophisticated research priority discussion between stakeholders

In a simple case, the research priorities as illustrated in Diagram 1 can be determined by the staff within an organisation itself. However, where an organisation is embedded inside a complex sector with other agencies above, alongside and below it, and a plethora of community and academic players are involved, there needs to be mechanisms for developing research priorities which involve a range of stakeholders. Diagram 2 sets out a representation of such stakeholder discussion:

Diagram 2: Strategic discussion between stakeholders on research and evaluation priorities

On the right-hand side of Diagram 2 there is the intervention logic linking policies, programmes and activities to high-level outcomes, this should be reflected in the organisation’s Statement of Intent. In the box on the left-hand side there are the stakeholders who should be involved in regular (e.g. annual) discussions. These discussions should:

· Look at research and evaluation findings which have come in since the previous discussion

· Identify priority research and evaluation questions

· Identify what research and evaluation the stakeholders have already planned for the next period

· Work out how priority research and evaluation will be funded and carried out.

· Identify issues of research and evaluation capacity for the sector and initiate appropriate action to improve these.

Ensuring that such discussions take place and are fruitful is not a trivial exercise and requires significant resources. At the current time various sectors undertake some aspects of such discussions. Such discussions need to focus on research priorities rather than on immediate resourcing or delivery issues, this may not always be easy to achieve when facilitating such discussions. Such discussions need to:

· Be supported by a knowledge management infrastructure which allows an overview of research and evaluation projects and findings in the last period in a way that allows a rich and sophisticated discussion to take place around priorities.

· Be undertaken by stakeholder representatives who understand what is possible in research and evaluation. Stakeholder representatives need to have the capacity to: identify the key research and evaluation questions which need to be answered; identify the possibilities of answering these questions from a methodological point of view; an ability to assess the likely relative cost of alternative research and evaluation projects asking different questions; and an ability to assess the likelihood of proposed projects actually being seen through to completion given the realities of implementing research and evaluation projects on the ground in the sector. This work will need to involve research and evaluation strategic and methodological “interpreters” to assist discussions around priority setting.

· Be a developmental process. It is likely that the first few iterations of such discussions will need to focus on building the capacity to have such discussions rather just focusing on providing a list of priorities.

2.3 Technical characteristics of intervention logics for use in linking R&E

There are technical characteristics of intervention logic(s) used for linking research and evaluation priorities within an SOI. These characteristics affect the ease with which they can be used for linking research and evaluation strategy. These technical issues have tended to not be well understood in the SOI process as it has developed so far, but they become particularly important when thinking about using an organisation’s Statement of Intent as a basis for research and evaluation planning, particularly when the organisation is only one actor in a complex wider sector. The issues are: the type of intervention logic being developed; the ease of making evidential links in an intervention logic; and the constraints placed upon what will be accepted as a well-formed outcome in such logics. These are discussed below.

2.3.1 The type of intervention logic

There is not just one single type of intervention logic. The different intervention logics which can be developed for a project, organisation or sector include:

· the intervention logic as imagined by those who planned the intervention (the planned intervention logic)

· the intervention logic as supported by the existing research base (research evidence-based intervention logic)

· the intervention logic as implemented (implemented intervention logic).

In a learning organisation these intervention logics should be connected as set out in Diagram 3.

Diagram 3: Connection between types of intervention logic in a learning organisation

2.3.2 Ease of making evidential links in an intervention logic

When using intervention logics in a learning organisation as described above, it must be realised that different types of interventions differ in their ease of evaluability. This means that a simplistic approach cannot be taken to the development of the research evidence-based intervention logic illustrated in Diagram 3. Diagram 4 sets out what is normally being attempted in an evidence-based approach – using evidence to link interventions (programmes or outputs in this diagram) to outcomes. If this can be achieved then it obviously provides a powerful tool for developing improved policies and programmes.

Diagram 4: The evidence-based attempt at linking outcomes and programmes/outputs

However interventions (programmes, activities and policies) differ in regard to their ease of evaluability. Relatively easy-to-evaluate interventions will tend to:

· operate at only the individual level rather than include organisational, community and policy-level strategies

· take place in only one locality rather than at the national, in addition to the local, level

· focus on single-outcome variables that are already routinely collected, rather than multiple-outcome variables

· take place in institutionalised controlled settings

· seek outcomes that can be measured within a relatively short timeframe.

For instance, a school-based programme that uses examination results as its outcome measure is a good example of where it is relatively easy to measure and attribute changes in outcomes to the effects of a programme.

In contrast to such programmes, many of the interventions in complex sectors in an age of “joined-up solutions” have characteristics which mean that they are more difficult to evaluate. They tend to:

· use a range of strategies at the individual, community and policy level in an integrated programme

· take place at both the local and national level at the same time

· be directed at multiple rather than single outcomes, some of which may be expensive to collect data on

· take place in uncontrolled community, rather than institutional, settings

· seek long-term outcomes that will take years to come to fruition.

In these cases outcome evaluation is much more difficult. This means that a simplistic approach cannot be taken to developing the research evidence-based intervention logic. Diagram 5 outlines the complexity of this type of situation.

Diagram 5: Ease of evaluability of link between outcomes and programmes/outputs

Diagram 5 represents a situation with interventions which have different ease of evaluability. On the left-hand side of Diagram 5, it may be possible to obtain clear outcome evaluation results to attribute the outcome of a reduction in offending to small group programmes run by the Department of Corrections. However it is also likely that capacity building programmes, funded by Te Puni Kokiri, working at the community level (and hence harder to evaluate for attribution of outcomes) may be also be contributing to reduced offending. Equally, a community action programme funded through the Health Vote that includes an element of reducing alcohol abuse may also reduce offending.

Another example is provided on the right-hand side of the diagram, where a reduction in road accidents can be directly attributed through evaluation to Land Transport Safety Authority activity. However, a community action programme from the Health Vote may also reduce road accidents due to its targeting of alcohol abuse (but again, because it is a community-based strategy it is likely to be more difficult to provide outcome evaluation evidence to directly attribute reduced road accidents at the same level of certainty as can be done for Land Transport Safety Authority activity). This example illustrates the fact that a high level of sophistication is needed in developing research evidence-based intervention logics and the research and evaluation strategies which underpin them.

2.3.3 Constraints upon what will be accepted as well-formed outcomes within intervention logics

A further technical aspect which needs to be considered when thinking about the link between an organisation’s research and evaluation strategy and its SOI relates to the notion of outcomes (either high-level or intermediate) within intervention logics. An analytical framework dealing with the principles of outcomes systems is outlined in more detail in Duignan (2004). When examining any particular intervention logic it is important to understand what is, and what is not, acceptable as a well-formed outcome. A useful way of viewing this is to think in terms of the constraints which are being on what is acceptable as a well-formed outcome within a particular intervention logic.

Constraints can be demanded for an entity to be accepted as a well-formed outcome in regard to the following:

· Measurability - the extent to which it is possible and feasible (within financial, time and ethical constraints) to measure the outcome

· Attributability - the extent to which it is possible to attribute a change in an outcome to the activity of an organisation or person

· Accountability – the extent to which an organisation or person will receive rewards or sanctions as a consequence of a change in an outcome.

Outcomes hierarchies can look very different depending on the constraints which have been placed on the outcomes which are used within them. For instance Diagram 6 shows an unconstrained intervention logic which is not constrained in terms of measurability, attributability or accountability.

Diagram 6 An unconstrained intervention logic unconstrained in terms of measurability, attributability or accountability.

In Diagram 6 the small circles indicate those intermediate outcomes for which evidence can be provided that attributes any change in the outcome to the activities of the organisation concerned. As is often the case in a sector with many actors and programme and which is affected by many other variables, this diagram shows the attributable intermediate outcomes to be few and at a low level.

Diagram 7 in contrast to Diagram 6, shows the intervention logic which only includes as intermediate outcomes those which survive the imposition of the constraints of measurability, attributability and accountability.

Diagram 7 A constrained intervention logic with the constraints of measurability, attributability and accountability

These two types of intervention logic serve very different purposes. The first, unconstrained logic (Diagram 6), is most useful for strategic planning purposes. Because it is not constrained by measurability, attributability or accountability, it can be used to ensure that the organisation focuses on the outcomes chain right through to the highest-level outcome in terms of what is trying to be achieved.

The second, constrained intervention logic (Diagram 7) is most useful for performance management in that it consists of outcomes which can be measured and attributed to the activity of the organisation and for which it is to be held accountable. It is often of little use for strategic planning in that it just focuses on what is it that an organisation is trying to achieve, which is also measurable and attributable to its actions and for which it is accountable. In a complex sector with many actors, programmes and factors affecting outcomes, and with limited resources for evaluation, measurable and attributable outcomes are likely to be a small subset of the intermediate outcomes which are being sought.

What this means for developing a research and evaluation strategy based on a SOI is as follows:

· For a research and evaluation strategy to be comprehensive it needs to be linked to an intervention logic which is not constrained by measurability, attributability or accountability. This is because if it is linked to an intervention logic which is constrained by any of these it will not be providing the research and evaluation necessary to inform the broad strategic decisions which need to be made by the organisation. Such an unconstrained, comprehensive, intervention logic will include all of the intermediate and high-level organisational, sector and wider government societal outcomes to which the organisation is contributing. It will also provide the basis for a rich conversation with external stakeholders about research and evaluation activity and assist in defining research and evaluation priorities for the sector and flowing from these, research and evaluation priorities for the organisation itself.

· A sub-set of a research and evaluation strategy can provide information to assist in the use of a constrained intervention logic for performance management purposes.

3 Measurement and evaluation strategy and the SOI

A research and evaluation strategy has a key role in informing the measurement and evaluation work which need to underpin any SOI. In thinking about measurement and evaluation in regard to SOIs, it is important that the way in which such measurement and evaluation is conceptualised deals with the issues of attributability and accountability. One way of doing this is to identify three levels of measurement and evaluation. Depending on the way in an intervention logic within an SOI has been set up, it is sometimes possible to see these three levels as corresponding to the top, middle and bottom of the logic/hierarchy.
High level outcome measurement

The first level of measurement is of high level outcomes which need to be measured to see what progress is being made towards achieving high-level outcomes. For many organisations, changes in such outcomes are difficult to attribute to just the work of the organisation itself and involve the activity of a number of other players. It is therefore not essential that such measurement attributes any change in the outcome to the work of the organisation. If it can be done that is good, but if measurement is limited to those outcomes that can be attributed to the particular organisation then it is likely that important measures will be missed which allow tracking of whether the high level outcomes are being achieved, independent of the issue of attribution.

3.1.1 Selective evaluation studies

The second level consists of selected (because of feasibility and resource constraints) priority evaluation studies which look at aspects of the intervention logic. These have three potential purposes:

1. to further improve the intervention logic (e.g. a literature review about possible ways of intervening to bring about change in an area)

2. studies to investigate what is actually happening in an area and what could be improved

3. attribution of changes in outcomes resulting from the activity of the organisation (usually only changes in intermediate outcomes and highly selective due to feasibility and cost considerations).

3.1.2 Routine attributable performance monitoring

The third level is the measurement of routinely measured attributable performance measures. These are measures which can be cheaply and regularly collected and equally importantly, attributed to work of the organisation itself. They tend to be “output” type measures but are not restricted to currently defined outputs for an organisation. They should be linked to the constrained (in terms of measurement, attribution and accountability) intervention logic and should be as high up the logic (in terms of outcomes) as possible.

4 Summary of general principles in linking R&E to an SOI

The previous sections have set out the general principles for linking a R&E strategy to an SOI. In summary, it is not difficult to link research and evaluation to an SOI as long as the following apply:

Nature of the SOI and its underpinning intervention logic
The intervention logic underlying the SOI has been set up as an unconstrained comprehensive intervention logic so that it can provide a broad strategic framework for research and evaluation planning (as set out in Section 3.3.3). The research and evaluation strategy can still contribute to complementary constrained intervention logics which are limited to measurable, attributable and accountable outcomes developed for performance monitoring purposes.
Stakeholder research and evaluation priorities discussion
For an organisation in a sector involving multiple actors, programmes and factors affecting outcomes, it is important that a rich strategic discussion takes place amongst stakeholders regarding research priorities and activities (as set out in Section 3.2).
Knowledge management infrastructure
An appropriate knowledge management infrastructure needs to be put in place to manage information about research projects and findings for the strategic discussion described above and for generally managing the information related to the research and evaluation strategy.

Research and evaluation capacity building to enable identification of priorities
Research and evaluation capacity needs to be increased in a number of areas. It needs to be built in regard to undertaking research and evaluation projects and building support within an organisation’s culture for research and evaluation (which is not dealt with in this paper). Focusing specifically on linking R&E to an SOI, capacity firstly needs to be built in the case of internal staff to ensure that they can undertake the task of identifying research and evaluation priorities. Secondly, it needs to be built to enable stakeholders to undertake a sophisticated discussion of research priorities. For these two areas those involved (from the research and evaluation planning staff to the highest levels in the organisation and external stakeholders) need to have the capacity to: identify the key research and evaluation questions which need to be answered; identify the possibilities of answering these questions from a methodological point of view; an ability to assess the likely relative cost of alternative research and evaluation projects asking different questions; and an ability to assess the likelihood of proposed projects actually being seen through to completion given the realities of implementing research and evaluation projects on the ground. This work will need to involve research and evaluation strategic and methodological “interpreters” to assist discussions around priority setting.
Three levels of measurement and evaluation within an SOI
In thinking about research and evaluation strategy related to an SOI, there needs to be clarity in the measurement and evaluation approach regarding attribution and accountability. This can be dealt with in using a three level approach to measurement and evaluation. First, high-level measurement of outcomes for strategic purposes to ensure that the overall strategy, involving many sector players, is on track. Second, selective priority evaluation studies looking at aspects of the intervention logic. Third, routine performance measurement of easily measurable and attributable indicators (often at the output level) for which the organisation is accountable.

5 Conclusion

This paper has discussed some technical aspects of linking research and evaluation to an organisation’s Statement of Intent. It has set out the Research Evaluation Monitoring Intervention Logic Outcomes methodology (REMLogic Outcomes methodology) as a way of doing this. The development of SOIs within the context of a managing for outcomes approach has great potential for assisting in organisational planning for research and evaluation. As long as the technical issues outlined in this paper are taking into consideration the development of SOIs should foster better focused research and evaluation.

6 References

Duignan, P. (2004, June 3). Principles of Outcome Hierarchies: Contribution Towards a General Conceptual Framework for Outcomes Systems (Outcomes Theory) [WWW document]. From The Strategic Evaluation Web Site. URL http://www.strategicevaluation.info/se/documents/122f.html.

¹ Dr Paul Duignan works for Parker Duignan Ltd Wellington and is a Senior Research Fellow at the Centre for Social and Health Outcomes Massey University Auckland, New Zealand (SHORE).

² The intervention logics used in this paper illustrate only one of several ways in which intervention logics can be laid out. This type of intervention logic can be called an outcomes hierarchy backbone intervention logic. Various other methods, including complex tabular approaches can be used in intervention logic work. Note also that in this intervention logic outcomes are only linked to the immediately higher outcome level, in a number of such logics there will also be more complex connections which may skip or work within levels.

Duignan, P. (2004, June 3). Linking Research and Evaluation Plans to an Organisation's Statement of Intent (SOI) [WWW document]. From The Strategic Evaluation Web Site. URL http://www.strategicevaluation.info/se/documents/120pdff.html.

Dr Paul Duignan1

2 Actually Linking R&E with an SOI

4 Summary of general principles in linking R&E to an SOI

Duignan, P. (2004, June 3). Principles of Outcome Hierarchies: Contribution Towards a General Conceptual Framework for Outcomes Systems (Outcomes Theory) [WWW document]. From The Strategic Evaluation Web Site. URL http://www.strategicevaluation.info/se/documents/122f.html.

Dr Paul Duignan¹