Measuring Public Sector Performance 2001 Conference

Wellington, 16-17 October 2001

Dr Paul Duignan

(Updated Version 2.1 25-10-01)

This paper is able to be accessed at www.parkerduignan.com/documents/101.htm

Introduction

Continuing Outcomes Emphasis

The increasing emphasis on outcomes in public policy is continuing as we move into the 2000’s and the public sector attempts to reinsert outcomes into the mechanisms for public policy formation and implementation. This follows over a decade or so of outcomes neglect due to the way the public sector 1980-90’s reforms were implemented.

Most of the stakeholders are now reattuned to the importance of outcomes. It is important to get right exactly how and where any new outcomes-orientated mechanisms are inserted into public sector accountability and strategic planning processes. It we get this wrong we stand to waste a lot of time as we demand new “outcomes-orientated” ways of agencies reporting on their activity, but without necessarily creating any more leverage on actually getting public sector programmes to lead to outcomes more quickly and efficiently.

Two particular problems are, first, a lack of understanding of the formal characteristics of outcomes, outputs, agency objectives and related frameworks and sets. For instance, outcomes are a characteristic of the broader world outside of an individual agency. An individual agency usually (although not in the case of all programmes) only makes a partial contribution towards the achievement of such objectives. For broader outcomes it often does not make sense to demand that an individual agency work towards a unique set of “outcomes” rather than outputs, or some other (currently somewhat maligned) “process” level set of objectives. An agency can, of course, contribute towards wider outcomes, however immediately the term contribute towards enters into an agency’s objective statement, the question of attribution and accountability for an individual agency achieving that objective becomes an issue. Formal issues related to outcomes, outputs and objectives such as these seem to be little understood.

Secondly, there continues to be extreme optimism as to the ease with which we will be able to attribute changes in outcomes to particular programmes, policies or agencies. This is much more difficult that is currently thought, particularly in most of the high profile cross-sector social policy areas which are currently being focused on. Even where such attribution is technically possible it is, for a country the size of New Zealand, often unaffordable.

It would be unfortunate to lose some of the most useful elements in the previous reforms, by too hasty misguided attempts at changing structures in order to refocus on outcomes. In my view the problem with the current system is not so much a structural problem (although there have been problems in its practical application) but an almost total neglect of anyone seriously taking up the task of linking outputs and outcomes during the 1980-90’s public sector reforms. There was provision for this linking activity in the structure of the 1980-90’s reforms, but it was simply not done to any significant degree. Perhaps the most fruitful approach at the moment is to increase such activity without drastically changing the current performance management framework.

We are still not making as much progress as we should be in sector evaluation priority setting. Programme and policy evaluation is still, in many cases, continues to be looked at solely from the point of view of an individual programme or policy. In addition to taking into account the characteristics of an individual programme, priority evaluation questions also need to be determined from the point of view of a sector seeking to answer key questions for feeding into its strategic planning processes. There is little guidance currently flowing through to evaluators who are designing evaluations for particular programmes as to what are the priority questions to be answered in the evaluation of a particular programme. This is apart from a general, often misguided, hope that the evaluation is going to be able to definitively attribute changes in outcomes to the programme. There are however, some initiatives starting in this area, for instance in the labour and employment area, where steps are being taken to develop a more coordinated approach to evaluation priorities. In addition, the work being done by the Department of Corrections systematically examining different types of interventions and the work by the Land Transport Safety Authority work on road safety are examples of more integrated evaluation approaches being driven from a strategic perspective.

A considerable number of evaluations of social programmes have been undertaken in New Zealand. These contain a large number of insights into the programmes which have been evaluated and which could be useful for planning future programmes. However, there is a lack of formal, and it seems, inadequate informal mechanisms to ensure the use of previous evaluation results are used in the planning of new programmes. This is also reflected in the lack of learnings being passed from one sector to another.

The fact that governments, of whatever political persuasion, continue to demand, what are in my view totally inadequate lead and planning times for programmes adds to the lack of attention to previous evaluation findings. There is simply no time for those designing and implementing such programmes to go back through previous evaluation findings to ensure that programmes are optimised.

It often may be better to invest evaluation money into bringing together of previous evaluation findings together with the proposed programme logic of a programme, rather than just commissioning another attempt at outcome evaluation. Such evaluation processes are known in the evaluation literature as Prospective Evaluation Syntheses.

The lack of planning and inadequate processes for the effective implementation of programmes points to the need for formative evaluation. Formative evaluation is concerned with ensuring that programmes are well formed. There are various models for formative evaluation, for instance, having an independent evaluation team working alongside a programme as it develops. As an example, the Alcohol and Public Health Research Unit at the University of Auckland was funded to play a formative evaluation role alongside the Heartbeat New Zealand project. This consisted of working with the project as a ‘critical friend’. The development of the project was monitored, assistance provided with tasks such as objective and strategy setting, piloting, and developing evaluation measures. The Unit presented a written report to the project and its funder each quarter raising issues and suggesting ways of proceeding.

Formative evaluation should be required of major projects to ensure that they are adequately planned and implemented. From an evaluator’s perspective such formative evaluation should always be in place for major programmes that are being planned. This process would help to ensure that public servants were being given adequate time to plan and develop programmes and assure politicians and tax payers that programmes were being carefully developed rather than just rushed to meet an urgent political agenda.

As discussed in the attached paper, we need to continue to be clear about the relationship between performance management and evaluation. This is set out in the diagram below and explained more fully in the paper.

Evaluation findings should feed into (they do not currently do so adequately) the strategic conversation between Ministers, policy agencies, providers and ideally community stakeholders. This discussion should generate the outputs, policies and programmes for the next period. Performance measures (within the power of the programme to achieve and able to be attributed to the programme) should then be set and monitored. The next part of the strategic conversation should be to set evaluation priorities for the subsequent period so that the subsequent round of strategic planning can be better informed.

We need to work on a number of fronts to ensure that the strategic discussion focuses on outcomes and the link between these and outputs, programmes and policies. There are several initiatives being undertaken which hopefully will facilitate a richer strategic discussion of outcomes, intervention logics and the relationship to outputs, programmes and policies. The State Services Commission’s work on new reporting procedures for agencies should make the discussion between Ministers, central agencies and agencies more focused and coherent. Roger Waite’s work on the Pathfinder Project which he is discussing at this conference, will hopefully create the basis for a richer strategic discussion. In addition, the publication of The Social Report 2001 by the Ministry of Social Policy (Policy 2001) can feed into this discussion, along with other agencies initiatives around outcome measurement. Also the work around outcomes for geographical communities being undertaken by Local Government some of which is reported on by Leigh Gatt at this conference, can provide a regional perspective to the strategic discussion.

We need to work out how to include community stakeholders in the national strategic discussion to obtain their views regarding both policy and programme direction and also to obtain provider perspectives. In my view, Wellington is still not doing this adequately. There is considerable talk and some funding for autonomous ‘bottom-up’ community initiatives, but a lack of mechanisms or even perspectives in Wellington that the community needs to be engaged in all levels of the strategic discussion, rather than being theorised as a grass-roots player ‘out there’ somewhere in the regions.

Following the trend in medicine and related disciplines, we are now seeing strong calls for evidence based policies and programmes within all sectors of Government. The call for evidence based policy is obviously desirable since it is likely to lead to more efficiently addressing the needs of the community. However, we have to be realistic about the evidential model we are going to be able to build. Particularly if we want to look at the effect of policies in different sectors on cross-sector outcomes. As discussed in the attached paper, I believe that the evidential model we are going to build is going to look very patchy for at least the foreseeable future. Given that there are major differences in the ease with which outcomes evaluations and attribution can be made in different sectors, there is a great danger of ‘instrument bias’ where policy instruments are preferred which are easy to evaluate, rather than those that are most likely to be successful. I will watch with interest the development of the Pathfinder Project to see how it is proposed to deal with this issue in what appears to me to be an attempt to develop a very comprehensive evidential model.

The early history of evaluation (in the United States in the 1970’s) held high hopes for the experimental society (Campbell 1975). The idea that we could apply rational scientific experimental techniques to work out the best methods for solving social problems and dealing with social issues. This is still the unexamined model behind much of the current discourse about outcomes and evaluation in New Zealand. This model obviously fits some circumstances, where programmes can be turned on and off and where small scale experimentation can take place. Such programmes tend to be those which function at the small group or individual level and can be put into and pulled out of communities if they are shown not to work. However, many of the recent initiatives in social policy are attempting to work on multiple outcomes right across all groups within a community. The experimental paradigm does not sit well with this kind of approach. Communities do not appreciate being ‘experimented on’ and often simply reject the traditional types of evaluation design. We need to look at the assumptions behind how we conceptualise our work on outcomes and evaluation. What we have learnt regarding evaluation in the health promotion sector (which has been working with cross-sector community based programmes for a decade or so) is that we are often trying to spark ‘mini social movements’. These are all about raising morale and getting communities working actively towards their goals. Such approaches do not lend themselves to ‘experimentation’ regarding the programme. In one sense they cannot afford to fail. There may be attempts at long-term outcome measurement of such programmes, but due to the diversity of communities, the timeframes, the practicality and cost of such endeavours, there are probably better ways of using evaluation resources.

Many of the social programmes currently being promoted in New Zealand, whether mainstream, or Maori for Maori, are seeking to improve outcomes for Maori. This raises fascinating questions for evaluation. What evaluation approaches are useful for such programmes and acceptable to Maori stakeholders? What are the implications of the Treaty of Waitangi in terms of evaluation? It is clear that we cannot just assume that all current Western evaluation models will work in Maori contexts. The author is currently involved in a project funded by Te Puni Kokiri that is looking separately at government expectations of evaluation of Maori programmes and Maori perspectives on evaluation models.

Evaluating evaluation is a difficult task. It was originally assumed that evaluation results would simply be adopted by policy makers and policies changed on the basis of whatever the evaluation results showed. In practice the effect of evaluation on policy making is much more diffuse than a direct connection with policy formation. Carol Weiss has termed this the ‘enlightenment’ function of evaluation (Weiss 1999). This is the ‘percolation of new information, ideas and perspectives into the arenas in which decisions are made.’ However, it is very difficult to identify the exact effect an evaluation has in terms of such enlightenment. As Weiss says: ‘the slow trickle of enlightenment is hard to see and harder still to identify as the product of evaluation’.

What steps can we taken then to evaluate evaluation? Some steps could be as follows: