Principles of Outcome Hierarchies:
Contribution Towards a General Analytical Framework for Outcomes systems (Outcomes Theory)
Strategic Evaluation Working Paper
Dr Paul Duignan1
Parker Duignan Ltd
Comments on this paper gratefully received to:paul@parkerduignan.com
Duignan, P. (2004, June 3). Principles of Outcome Hierarchies: Contribution Towards a General Conceptual Framework for Outcomes Systems (Outcomes Theory) [WWW document]. From The Strategic Evaluation Web Site. URL http://www.strategicevaluation.info/se/documents/122pdff.html.
V3-1-10 22-6-04
If using models from this paper in any context please source them to this document, the reference format is on the front cover.
Introduction
This paper is a draft technical working paper for those with an interest in understanding, developing or using outcome hierarchies within outcome systems. It is an initial contribution towards outcomes theory – a general analytical framework for outcomes systems which is progressively being developed by the author. An outcome hierarchy is a structured set of all of the important intermediate outcomes with lead through to a final outcome being sought by an organisation or group of organisations. The paper starts with a set of definitions as used in the author’s theory of outcomes. It then goes on to outline a set of principles of outcomes theory as they relate to outcome hierarchies. The area of outcome systems and Managing for Outcomes is currently under theorised. Many of the terms used by those working with such systems are used in ambiguous ways. The purpose of this paper is to assist clearer discussion about outcome systems and better formulation of outcome hierarchies and other elements used in such systems. The definitions and the principles in this paper are all provisional and any comments, suggested improvements, amendments or concrete examples of outcome sets which appear to have violated these principles would be appreciated by the author.
Definitions
Organisation
Any entity from an individual upwards which undertakes purposeful activity. A higher-level organisation is one which directs a lower-level organisation to undertake specific activity or activities or to attempt to bring about change in an outcome or outcomes.
Outcome
An outcome is not a natural entity with a fixed definition2. What qualifies as an outcome in any outcome system depends on the demands made by those who are implementing the system. The most useful way of conceptualising outcomes is to characterise them in terms of the constraints a particular outcome system requires for a statement to be regarded as a well-formed outcome. Set out below is a set of definitions with progressively more demanding constraints on what is acceptable as a well-formed outcome within a particular outcome system.
- A variable3 which it is hoped to change in a particular direction.
- A variable which it is believed may theoretically be able to be changed by an organisation’s activity.
- A variable as defined in 1 or 2 above which is external to in some sense rather than internal to the organisation undertaking the activity.
- A variable as specified in 1, 2, or 3 above for which change can be measured (this can be further constrained by any of the levels of measurability set out in the definition of measurability below).
- A variable as defined in 4 above for which change, or lack of change, can be attributed to the activity of an organisation (this can be further constrained by any of the levels of attribution set out in the attributability definition below).
- A variable which has selected additional constraints put on it. The most well known list of these constraints is the SMART acronym for outcomes. SMART4 stands for Specific, Measurable, Action-orientated, Realistic and Time-specified.
When designing an outcome system it is essential that system designers specify the constraints being placed on what will be accepted as a well-formed outcome within the system. If this is not done there will usually be confusion amongst those implementing the system as to what is and what is not an outcome and great variability in the outcome sets which are developed within such a system.
Output
Best viewed as a special case of an outcome which has the constraints that it is routinely measurable at low expense and totally attributable to the action of a single organisation.
Measurability
The extent to which an outcome is measurable. Outcomes differ in how technically and practically difficult it is, and how much it costs, to measure them. The degree of measurability of any outcome can be described as one of the following: impossible, prohibitively expensive, expensive, relatively inexpensive, or routine. Measurability is very often confused with attributability.
Attributability
The extent to which a change, or no change, in an outcome is able to be attributed to the action of an organisation with a high degree of certainty. Sometimes this is talked about in terms of the impact of a programme (discussed next). Attribution is usually much more difficult to achieve than measurability. Measurability is a necessary, but not sufficient requirement for attribution. Attribution (apart from those cases where it is self-evident as in the case of outputs) usually requires some sort of experimental or quasi-experimental evaluation design. Attributability of a change, or no change, in an outcome to an organisation can be described as one of the following: impossible, prohibitively expensive, expensive, relatively inexpensive, or routine.
Impact
Impact is the effect a programme has on an outcome. It is a term often used in outcome systems and sometimes contrasted with the term outcome. The term impact is not currently used as a formal term in the approach developed in this paper. This is because the term on its own is unclear as to whether it refers to the theoretical impact that a programme could have prior to commencement; the actual impact a programme has (regardless of one’s ability to actually empirically attribute this to the programme); the feasibly attributable impact a programme has; or the actually attributed impact a programme has been established to have had. This lack of clarity renders the term less than useful in a number of contexts. In the context of this paper an impact is best viewed as a type of outcome which has various attributability constraints placed on it. Of course, in the specification of an actual outcome system, it is possible for designers to use the term impact for outcomes with a particular set of attributability constraints as long as they have clearly specified those constraints.
Accountability
The extent to which an organisation is held accountable for producing or not producing a change in an outcome. Accountability relies on both measurability and attributability. Accountability always needs to take into account whether an organisation has autonomy in selecting strategies for changing an outcome.
Autonomy in strategy selection
The extent to which a lower-level organisation is free to select from the full range of possible strategies for bringing about change in an outcome.
Outcome Domain
The area or sector in which an outcome is located. For instance, health, trade, education, welfare.
Intervention logic
The setting out, and often visualisation of, the steps, stages, intermediate outcomes, and sometimes other elements (activities, outputs) which is it believed are needed in order to achieve change in a final outcome in a domain. There are a range of different tabular and diagrammatic ways of setting out an intervention logic.
Outcome hierarchy
A specific type of intervention logic which only includes outcomes. It sets out all of the intermediate outcomes it is believe are necessary in order to achieve a change in a final outcome. For the purpose of this discussion it is assumed that such hierarchies are visualised in two dimensions and allow for interlinkages between intermediate outcomes across levels.
Strategies
Ways of going about achieving an outcome. How an organisation is going to do something rather than why it is doing it (the outcome). Looking at it this way, a strategy can be seen as the how and an outcome as the why. From a wording point of view, a strategy can be changed into an outcome by changing it into a noun and an outcome can be changed into a strategy by changing it into a verb (e.g. a strategy of reducing unemployment can be translated into an outcome of reduced unemployment).
Indicators
Measurements of variables which are used to attempt to measure achievement of an intermediate or final outcome. If an outcome is conceptualised as an underlying construct variable, then an indicator is a measure of it. Looking at it this way there may be more than one indicator for an outcome.
Performance indicators5
Measurements of variables for which change, or lack of change, can be attributed to an organisation.
Targets
Values (or levels) on variables used as indicators or performance indicators.
Benchmarks
Comparative values (or levels) on variables for performance indicators from other organisations.
Dose-response relationship attribution
Attribution which goes beyond just measuring whether or not the activity of an organisation affected an outcome to show how much of an intervention is needed to affect an outcome to a particular level.
Principles of Outcome Hierarchies
A set of principles are set out here regarding outcome hierarchies as a sub-set of intervention logics. Due to the diversity of types of intervention logic it would not be possible to specify a set of technical principles for them in the same level of detail as is possible in regard to the much more specific concept of outcome hierarchies.
Principle 1: Outcome Hierarchies substitute for goal-mission-objective statements
A good outcome hierarchy should ultimately substitute for an organisation’s goal-mission-objective statement. During transition stages, goal-mission-objective statements may coexist alongside outcome hierarchies. However, a goal-mission-objective statement can be conceptualised as a simple outcome hierarchy which is limited to three elements (goal, mission, objectives) and does not allow cross relationships between lower level elements. In a visualised two dimensional outcome hierarchy, a richer set of relationships between intermediate outcomes can be examined, discussed and communicated. In spite of this, there may still be political or organisational reasons for maintaining goal-mission-objective statements.
Principle 2: There is more than one possible outcome structure for a domain
There is always some arbitrariness about outcome hierarchies in real world settings and so more than one possible outcome structure can potentially be developed for a domain. This is due to the fact that outcome hierarchies are normally constructed as two dimensional representations of multi-dimensional relationships. They could theoretically be modelled in three or more dimensions, but the level of complexity involved in interpreting such hierarchies means that usually two visual dimensions are used. There are usually a number of different ways of "cutting the cake" when developing an outcome hierarchy. Hierarchies tend to be the product of a number of pragmatic trade-offs related to the definition of the domain in which the hierarchy is being developed, the focus of the organisation developing the hierarchy and the purpose for which the hierarchy is to be used. Therefore there is not a single right or correct outcome hierarchy for a domain, just good or fit-for-purpose outcome hierarchies.
Principle 3: A strategy can always be converted into an outcome
An outcome is about an end to which some activity is directed (why). A strategy is about the way in which an outcome will be achieved (how). A strategy can always be converted into an outcome (by changing it into a noun) and an outcome into a strategy (by changing it into a verb). Depending on the way in which an outcome hierarchy is read it can look like a set of strategies or a set of outcomes. Looking up from any entity in an outcome hierarchy one can see the outcomes towards which one is directed. Looking down one can see the end points of the strategies one is using to achieve an outcome at any level.
Principle 4: Outcomes hierarchies are not logically limited to organisational boundaries
An outcomes hierarchy should not be essentially about an organisation, it is about a domain in the outside world. In the case of an organisation with a single outcome where the outcome is only influenced by the organisation alone, the organisational outcome hierarchy and the domain outcome hierarchy will match. In the case of organisations working with other organisations in a context where there are a range of other factors also influencing an outcome, the issue of how to structure an outcome hierarchy for use by a single organisation becomes more complex. One organisation will only be responsible for a sub-set of the strategies which are needed to change an outcome. Therefore, outcome hierarchies are likely to spread across organisational boundaries. For instance, if a government wishes to improve outcomes for a population group, a wide range of different organisations are likely to be involved in undertaking strategies to achieve this. Such cross-organisational schema are often described as sector strategies rather than outcome hierarchies. However, conceptually, there does not have to be a difference between these two. This is because a sector strategy needs to build off an analysis of the outcome hierarchy for a domain.
One approach to the situation where an organisation contributes to an outcome along with other organisations is for it to work collaboratively with the other organisations to develop a joint outcome hierarchy. Once this has been developed each organisation can then determines the intermediate outcomes it contributes to. This is essentially what occurs when teams or units within a single organisation come together to develop a joint outcome hierarchy and then go on to work out each teams' contribution to the hierarchy. Another approach is to develop a hierarchy for use by an organisation which is more about outcomes in terms of influencing other stakeholders than about the substantive outcome domain. This is often the case for organisations involved in policy development or coordinating other organisations. In these cases, it is useful to develop two outcomes hierarchies. The first is the substantive outcome domain hierarchy which can be used for strategic guidance (working in conjunction with other organisations), and the second is the narrower stakeholder influence outcome hierarchy for a single organisation.
Principle 5: Outcome hierarchies have more than one function
Potential functions of outcome hierarchies include the following:
1. For use in strategic planning. Outcome hierarchies provide a way of looking at all of the intermediate outcomes in a domain which need to be achieved in order to arrive at a final outcome. Visualisation of such hierarchies helps ensure that all of those involved in a strategic planning processes have a shared understanding of the outcome structure and the links and interactions in that structure. Once this shared understanding has been achieved, the next step in strategic planning is to work out what needs to be done to achieve the intermediate outcomes within the hierarchy.
2. For determining which indicators to measure in a particular domain. There are usually a wide range of indicators which can be measured in a particular outcome domain. An outcome hierarchy enables a rational selection of indicators based on measuring whether or not intermediate outcomes are being achieved. This assists in monitoring progress towards the achievement of a final outcome. It is useful, firstly, in cases where there is a long lag-time between achieving intermediate outcomes and achieving a final outcome. Secondly, in cases where it is difficult, expensive, or impossible to measure a final outcome and so intermediate outcome measurement can provide some assurance that final outcomes are likely to be improving (if the outcome hierarchy reflects the reality of the intermediate outcomes' relationship with the final outcome in the domain).
3. For accountability. Sometimes the attempt is made to use outcome hierarchies for accountability. This is based on the belief that intermediate outcomes will be easier to attribute to an organisation's activity than a final outcome. While this is likely to be true in a general sense, only attributable outcomes should ever be used for accountability. Just because an outcome is an intermediate outcome, this is no guarantee that it is actually attributable to the activity of a particular organisation.
Principle 6: Measurement does not mean attribution
Measurement and attribution are entirely different. Being able to measure an indicator says nothing about whether or not any change in the outcome measured by the indicator can be attributed to the activity of a particular organisation. In some instances, it is relatively easy to attribute changes in an outcome to particular organisations. In many other cases, it is extremely difficult. Even where it is theoretically possible to make attribution, it is often very expensive and hence such attribution cannot be used for routine accountability (say on an annual basis). In general, any organisation should have an outcome hierarchy for use for strategic planning. Such an outcome hierarchy, when being used for strategic planning, should not be constrained by any attributability requirements. If it is so constrained, then strategic planning will be skewed in the direction of the organisation doing that which it can prove is attributable to it alone. In contrast, an organisation should be encouraged to do what needs to be done in order to achieve the outcome it is charged with changing. It is appropriate to track a selected range of indicators flowing off an outcome hierarchy for strategic purposes. Again, there does not need to be attributability of these measures to the organisation in question. The purpose of tracking such indicators is to answer the question, does it seem that, over time, the outcome we are interested in is changing in the direction we want? Accountability is then dealt with as a separate issue.
In addition to an outcome hierarchy for strategic planning purposes, an organisation also needs a set of indicators (often called performance indicators, or key performance indicators (KPIs) and related to outputs) which should be totally attributable to the organisation for accountability purposes. In very simple cases, the indicators from the outcome hierarchy and the performance indicators may be the same. This is in instances where the outcome is affected by only one organisation, the indicators are cheaply measurable and totally attributable to the organisation concerned. In many real world situations the outcome hierarchy indicators and the performance indicators will be different because of the difficulty of attribution.
Principle 7: Attributional constraints on outcome hierarchies often result in "impoverished" hierarchies
As discussed in Principle 6, putting an attributability constraint on an outcome hierarchy is likely in many instances to lead to an "impoverished" outcome hierarchy. This is an outcome hierarchy which only captures a sub-set of the intermediate outcomes which are necessary to achieve an overall outcome. In this case the outcome hierarchy will only show the sub-set of intermediate outcomes which are able to be attributed to a particular organisation. This usually does not provide a useful tool for strategic planning in a domain.
Figure 1: An outcome hierarchy without attributional constraints (green or lighter) and with attributional constraints (purple or darker).
Figure 1 shows an outcome hierarchy without attributional constrains (the green, or ligher, ovals) and with attributional constraints (purple, or darker, ovals) The attributional outcome hierarchy is an impoverished version of the fully non-constrained outcome hierarchy. The impoverished attributional outcome hierarchy is illustrated in Figure 2. Such outcome hierarchies are generally useless for strategic planning purposes and are only useful for accountability purposes.
Figure 2: “Impoverished” outcome hierarchy due to enforcing attributibility constraint
Principle 8: Attributability only can be used for accountability if there is autonomy in strategy selection
Even where a change in, or lack of change in, an outcome can be attributed to an organisation, it does not necessarily follow that the organisation should be held accountable for change or lack of change in an outcome. An organisation should not be held accountable for an outcome where it does not have autonomy regarding the selection of strategies to achieve that outcome. For instance, a ministry may be charged by a government with improving outcomes for a particular sector of the population. The government cannot require accountability from that ministry for achieving improvement in the outcomes for the population group (even if attribution could be established) unless it gives autonomy to the ministry for the selection of its strategies. In the real world, governments almost always put numerous limitations on ministries regarding what they can, and cannot, do to bring about improvements in outcomes. For instance, at the simplest level, a ministry with a population sector focus is usually prevented from undertaking political advocacy for that population group.
Principle 9: Strategies do not necessarily need to be nested under an individual outcome
In many instances, organisations will be charged with achieving more than one outcome. In organisations which have more than one outcome, there is sometimes a tendency to attempt to nest strategies under individual outcomes. Such nesting is only appropriate where a strategy exclusively contributes to the achievement of one outcome but not any others. For most organisations a strategy will achieve more than one outcome. Therefore strategies do not have to be nested exclusively under a single outcome. However, in strategy development it is usually useful to start with one outcome and then determine what strategies will help achieve this outcome. As one works through each of the organisational outcomes it will become clear whether strategies are unique to individual outcomes or whether they span across outcomes. This often ends up just being a presentational issue. If an organisation has five outcomes and five strategies and four of the strategies are unique to a particular outcome while one strategy contributes to two outcomes, the best way to represent this is to present the strategies under each outcome and just repeat the one strategy which contributes to two outcomes. However if all five strategies contribute to all five outcomes, it would be more parsimonious to discuss the outcomes in general and then the strategies one at a time (not attempting to nest them under a under specific outcomes). The connection between the outcomes and the strategies can then be shown in a "wiring" diagram linking strategies with outcomes.
Principle 10: Sub-Units do not need to "see" themselves in a single organisational outcome
Sometimes there is the desire for sub-units within an organisation to be able to "see themselves" in a discrete high level outcome. It should be possible for a sub-unit to be able to clearly identify outcomes to which their activity contributes but they do not need to see themselves in a separate outcome. This is another form of Principle 9 that a strategy does not need to be "nested" under an individual outcome. Equally, when looking across a group of organisations which are contributing to higher level outcomes, this principle also means that such organisations (individual government ministries for example) do not have to “see" themselves in discrete high level outcomes, rather they must be able to see high level outcomes to which they contribute.
Principle 11: Lower-level organisations do not necessarily need to have single integrated outcome structures
In some instances, it is appropriate for an organisation to have a "non-integrated" outcomes hierarchy. A non-integrated outcomes hierarchy is one where the organisation has two or more independent outcome hierarchies which are not integrated at the top into an inclusive outcome. This can occur where a lower-level organisation has been directed to undertake one or more activities which relate to different outcome domains by a higher-level organisation. The higher-level organisation may have assigned this activity to the lower-level organisation for a range of reasons unrelated to how closely related the two outcomes are. The classic instance of this is the case of the "catch-all" government department which undertakes a diverse range of activities which are too small to stand alone. In this case, it is not appropriate for a department to attempt to "fit" all of its activity under a single outcome hierarchy. It should develop different outcome hierarchies for different activities directed at outcomes in different domains. It is perfectly possible that such "stand-alone" outcome hierarchies are more closely related to activity taking place in another organisation, than with activity taking place within the host organisation. It is more useful to attempt to integrate outcome hierarchies with the hierarchies of outside organisation rather than to attempt to integrate them with other outcome hierarchies within the same organisation.
Figures 3-5 set out the possibilities where an organisation is required to undertake activity which is in two conceptually different outcome domains.
Figure 3: Organisation 1 is charged with undertaking activity directed at achieving outcomes in two conceptually distinct domains (green and purple or left and right within Organisation 1’s box).
Figure 4: An attempt to create a “hybrid” overall outcome to accommodate the two conceptually different outcome domains
Figure 5: Linking the conceptually distinct outcome hierarchy with an outcome hierarchy in the same domain from another organisation
Principle 12: Lower-level organisations must not over-extend their "outcome domains"
One of the purposes of developing outcome hierarchies is to assist in decision making about resource allocation to specific strategies. Higher-level organisations need to formally decide what domains they want grouped together in integrated outcome hierarchies. In a number of instances these groupings will not conform to organisational boundaries. If higher-level organisations are not clear about the need for them to make such decisions the following problem can arise. For a range of pragmatic reasons, a lower-level organisation is instructed to undertake a diverse range of activities by a higher-level organisation. The lower level organisation then groups all of these activities under a single integrated outcomes hierarchy. It proceeds to look at the top level of the outcome hierarchy it has constructed and to use this to make decisions about priorities for activity to achieve its outcome. It then decides that some of the activities which the higher-level organisation funds it for are not priorities in achieving its stated high-level outcome. This process is acceptable as long as the higher-level organisation has given the lower-level organisation discretion to decide on what activities it will and will not undertake. However, in many instances the lower-level organisation does not have such discretion and the reason it should continue to undertake an activity is not because it contributes to one overall organisational outcome but because it has been instructed to by its higher-level organisation as apart of a wider game-plan.
Principle 13: Resources can only be allocated to individual outcomes where strategies are exclusive to one outcome or "dose-response" attributability of strategies to outcomes has been achieved
The issue of attempting to allocate resources on the basis of individual outcomes comes back to the question of attributability. Usually resources are allocated on the basis of strategies or outputs (which in the approach in this paper are outcomes constrained by being easily measurable and attributable to an individual organisation). There are four instances where resources can be allocated on the basis of outcomes. The first is where the outcomes are easily measurable and attributable to a single organisation (as in the case of outputs). The second is where an organisation has only one outcome. In which case there is no additional information added by allocating resources on the basis of outcome. The third is where an organisation has multiple outcomes but its strategies are exclusive to individual outcomes. The fourth is where strategies are not exclusive to individual outcomes but where "dose-response" attribution has been achieved for strategies to outcomes.
Principle 14: Higher-level and lower-level organisations need to negotiate the amount of resource a lower-level organisation will put into outcome measurement
A higher-level and lower-level organisation need to negotiate the amount of resource which is put into outcome measurement and attribution by the lower-level organisation. This is for the following reasons:
1. The higher-level organisation may wish to undertake, coordinate, or have a different lower-level organisation undertake the measurement of outcome indicators or attribution of the lower-level organisation’s activity to changes in outcomes.
2. The higher-level organisation may be unreasonably demanding attribution for accountability purposes where it is not in fact feasible.
3. The lower-level organisation may not be undertaking feasible measurement and attribution in order to avoid having to face accountability for lack of changes in outcomes.
Summary
This paper has set out a number of principles of outcome hierarchies as an initial contribution towards the development of a general analytical framework for outcomes systems (outcomes theory). These are presented as the basis for further discussion rather than being definitive. It is likely that there are a number of other principles that it would also be worthwhile stating. The author has some which he is still formulating. The definitions and the principles in this paper are all provisional and any comments, suggested improvements, amendments or concrete examples of outcome sets which appear to have violated these principles would be appreciated by the author.
References
1 Dr Paul Duignan works for Parker Duignan Ltd Wellington and is a Senior Research Fellow at the Centre for Social and Health Outcomes Massey University Auckland (SHORE).
Acknowledgements: Comments on this paper from Marc De Boer, Sally Munro and Patrick Duignan have assisted in its preparation. In addition the author has had the opportunity to reflect on these issues through undertaking outcomes work for a number of New Zealand government agencies.
2 In some outcome systems the attempt is made to attach an absolute definition to the term outcome. For instance that it is a neutral measure of a state of being in contrast to a goal which is the preferred state of being. The approach adopted in this paper is to allow outcome a wide definition and simply talk in terms of the constraints which are put upon an outcome which allow it to be well-formed.
3 A variable can be defined for the purposes of this paper as a quantity or quality that can assume a set of values.
4There are a range of different terms attributed to the SMART acronym particularly in regard to the letter A within it. One alternative sometimes given to Action-orientated is Achievable, however this overlaps with Realistic. The measurability requirement of the SMART constraints is obviously the same as the measurability constraints identified in 4 above.
5 Performance indicators as used here could be called impact measures if the concept of impact as discussed above is used. Performance indicators are also often used in regard to the measurement of outputs. Within the approach being advanced in this paper this does not present a problem as outputs are just seen as outcomes with a certain set of constraints put on them. Within an outcome system, system designers may use the term performance indicator to only refer to a sub-section of what is being referred to as performance indicators in the approach outlined in this paper.