Building Social Policy Evaluation Capacity

 

6 September 2001

Paul Duignan PhD

Senior Lecturer

Alcohol & Public Health Research Unit

University of Auckland

Auckland, New Zealand

 

 

 

Draft of a Forthcoming Article to be Published in the Social Policy Journal of New Zealand in 2002.

Any comments on this paper gratefully received by the author. paul@parkerduignan.com

[Interim source for this draft http://www.parkerduignan.com/documents/100.htm]

 

 

This paper draws in part on the thinking behind a paper by the author on “Mainstreaming Evaluation or Building Evaluation Capability?” evaluation which won the American Evaluation Association’s (AEA) 2001 Conference President’s Prize.

Abstract

The last three years have seen an increasing interest in evaluation in the public sector in New Zealand. This trend could result in an adequately resourced and sophisticated approach to evaluation, involving policy and provider levels within government, Maori and third sector/community organisations. This in turn could lead to better formed and implemented social programmes and policies. On the other hand it is possible that unrealistic expectations, an unsophisticated model of evaluation, lack of strategic involvement of stakeholders and inadequate investment in appropriate evaluation capacity building, will result in the current wave of enthusiasm ultimately turning to disillusionment. It we use the current increased interest in evaluation to build and embed a sophisticated evaluation capacity right across the social policy sector we are likely to see a more positive outcome. Three key we need to take are: using appropriate evaluation models, including those appropriate for Maori programmes; building a sector culture of evaluation through appropriate evaluation training and awareness raising at all levels; and attempting to foster strategic sector-wide evaluation question priority setting.

 

 

Table of Contents

 

 

Abstract 2

Table of Contents. 3

Introduction. 4

Using an appropriate evaluation model 5

Demystifying evaluation. 6

Set of Evaluation Terms Which Apply Right Across the Programme Lifecycle. 8

Internal and external evaluators. 10

Methods for hard to evaluate real world programs. 11

Not privileging any one meta-approach to evaluation. 12

Sophisticated Model of the Evidential Base that Evaluation can Deliver 12

Evaluation Models for Evaluating Programmes for Maori 18

Building a Culture of Evaluation Through Appropriate Evaluation Skills Training at all Levels Within the Social Policy Sector 20

Organisational or sector level strategising to prioritise evaluation questions. 23

Conclusion. 27

Acknowledgements. 28

Author Note. 28

Bibliography. 28

 

Introduction

 

The final years of the last decade saw a mounting tide of interest in evaluation and an outcomes-focus within the New Zealand social policy community (Schick 1996; Bushnell 1998; Duignan 1999; State Services Commission 1999; Controller and Auditor-General 2000). From the point of view of the working evaluator, this seems to have also been accompanied by a significant rise in the amount of evaluation being funded and undertaken in New Zealand. It will be fascinating to watch how this develops over the next decade. If we are lucky it will result in more sophisticated evaluation being undertaken which will feed into the formation and implementation of better social policy. If we are unlucky there is likely to be an initial burst of evaluation activity for a few years with a lot of resources spent on elaborate technical evaluation designs. This may then be followed by a phase of disillusionment with evaluation due to unrealistic expectations as to what evaluation can deliver for social policy in New Zealand.

 

If we are to get the most out of the increased interest in evaluation we must build an enduring evaluation capacity in the social policy area. Part of this is increasing the number of evaluators involved with the sector as has been done in some evaluation capacity building (Compton, Baizerman et al. 2001) but it needs to go beyond this to put in place the following three elements:

 

1.      Using appropriate evaluation models

2.      Developing a culture of evaluation throughout the social policy sector by teaching evaluation skills appropriate for each level of the sector

3.      Sector level strategizing to identify priority evaluation questions, rather than just relying on evaluation planning at the individual programme level.  

 

Each of these needs to be undertaken while involving government, community organisations and Maori stakeholders in the development of a more strategic approach to social policy evaluation.

 

Using an appropriate evaluation model

 

 

Discussing an appropriate evaluation model may seem a slightly obscure and theoretical place to start in thinking about building social policy evaluation capacity. However, there are a number of different ways in which evaluation can be described and various models and typologies that are in use by evaluators (Cook and Campbell 1979; McClintock 1986; Patton 1986; Guba and Lincoln 1989; Rossi and Freeman 1989; Scriven 1991; Fetterman, Kaftarian et al. 1996; Chelimsky and Shadish 1997). From the author’s experience with these models and approaches, they are not all the same in terms of their suitability for social policy evaluation capacity building. Suitable evaluation models for capacity building should meet the following criteria: 

·        Attempt to demystify evaluation so that it can be understood and practiced at all levels within the social policy sector

·        Use a set of evaluation terms which emphasize that evaluation can take place right across a program’s lifecycle and is not limited to just outcome evaluation

·        Allow a role for both internal and external evaluators

·        Have methods for hard to evaluate, real world programmes, not just for ideal-type large scale expensive external evaluation designs

·        Not privilege any one meta-approach to evaluation (e.g. goal-free, empowerment)

·        Be based on a sophisticated understanding of what evaluation can actually deliver in terms of an evidence base for social policy

·        Take into account the need for evaluation approaches for evaluating Maori programmes that may be different from mainstream evaluation approaches.

 

Some evaluation models meet these criteria better than others. Each of the criteria is discussed below.

 

Demystifying evaluation

 

An appropriate evaluation model for social policy evaluation capability building needs to be able to be explained in clear terms to a wide range of different stakeholders with diverse training, backgrounds and experience right across government, Maori and the community sector. Such a model must at the same time be able to accommodate complex technical evaluation methodologies within the same easily understandable framework. 

 

One easy way to describe evaluation for capacity building is to conceptualise it as simply being about asking questions – of our programmes, organisations and policies. These questions are not something that evaluators alone should attempt to answer themselves; they are questions that should be an important concern of every policy maker, manager, staff member and programme participant. The high level question which the author uses in describing evaluation is the overall evaluation question:

 

·        Is this (organisational activity, policy or programme) being done in the best possible way?

 

This is then unpacked into a series of subsidiary questions:

 

·        How can we improve this organisation, programme or policy?

·        Can we describe what is happening in this organisation, programme or policy?

·        What have been the intended or unintended outcomes from this organisation, programme or policy?

 

A question-based introduction to evaluation helps to demystify the process of evaluation. It puts the responsibility for evaluation back where it belongs, on the policy makers, funders, managers, staff and programme participants to identify the questions they are interested in rather than leaving it solely with evaluators. It highlights that programme managers and staff cannot avoid these questions; they just have to work out ways of answering them. In most cases stakeholders will have to answer these questions through their own effort. However, in some instances they will need to call in specialised evaluation assistance. A question-based approach to evaluation is also well positioned to highlight the concept of sector level strategising about priority evaluation questions which is discussed later in this article. 

 

Set of Evaluation Terms Which Apply Right Across the Programme Lifecycle

 

In New Zealand at least, most stakeholders unfamiliar with evaluation still see it mainly in terms of outcome evaluation, although this narrow perspective is now starting to change. An appropriate set of terms for the different types of evaluation should highlight that evaluation consists of much more than just outcome evaluation. Two important evaluation dichotomies are often used to describe evaluation – the distinction between formative and summative evaluation and the distinction between process and outcome evaluation. Combining elements from both of these leads us to a three way typology which emphasizes that evaluation can take place right across the programme lifecycle, not just at the end – formative, process and impact/outcome. This is the three-way split used in the evaluation work of the Alcohol & Public Health Research Unit (Casswell and Duignan 1989; Casswell and Duignan 1989; Duignan 1990; Duignan and Casswell 1990; Duignan, Casswell et al. 1992; Duignan, Dehar et al. 1992; Turner, Dehar et al. 1992; Duignan 1997; Waa, Holibar et al. 1998; Casswell 1999; Health Research Council n.d.).

 

In this typology, which is based on the purpose for which evaluation is going to be used, formative evaluation (McClintock 1986; Dehar, Duignan et al. 1993; Tessmer 1993) is defined as: evaluation activity directed at optimising a program. (It can alternatively be described as design, developmental or implementation evaluation).

 

Process evaluation (Scheirer 1994) is defined in our typology as: describing and documenting what happens in the context and course of a programme to assist in understanding a programme, interpreting programme outcomes and/or to allow others to replicate the programme in the future. Note that this narrows the definition of process evaluation by not including the formative evaluation element.

 

Outcome evaluation (Cook and Campbell 1979) is defined in the typology as: assessing the positive and negative results of a programme. This includes all sorts of impact/outcome measurement, recognising that outcomes can be short, intermediate or long term and also arranged in structured hierarchies, e.g. individual level, community level, policy level

 

None of these terms are opposed to each other, they are seen as three essential purposes for evaluation. The three terms can in turn be directly related to the three subsidiary evaluation questions identified in the section above. They can also be related to the start, middle and end of a programme. This encourages thinking about how evaluation can be used right across a programme’s lifecycle, each type of evaluation – formative, process and impact/outcome – must be individually considered as a possibility for evaluation activity. If outcome evaluation proves too expensive or difficult there still may be useful questions that can be answered in regard to formative and process evaluation.

 

 

Diagram 1: The Relationship Between Types of Evaluation and Stages in the Programme Lifecycle

 

Internal and external evaluators

 

An appropriate evaluation model for building evaluation capability must also allow for the possibility of both internal and external evaluators (Mathison 1991; Minnett 1999). If evaluation is just seen as something that is only undertaken by external experts then there is little reason for internal staff to improve their evaluation skills.  This is particularly relevant for Maori and community sector organisations often with little access to outside evaluation resources. A useful evaluation model for capacity building needs to have plenty to offer the internal evaluator with limited resources for evaluation, rather than just focusing on the needs and concerns of the relatively well resourced, external evaluator. It is more likely that formative and process evaluation techniques will be the one which are possible within the usually limited resources available to internal evaluators. 

Methods for hard to evaluate real world programs

 

An appropriate evaluation model for capability building also needs to incorporate methods that can be used to evaluate a wide range of real world programmes which tend to present interesting evaluation challenges. One area where appropriate evaluation models are crucially important is in the area of community programmes. Evaluating community-based programmes presents interesting challenges for evaluators and raises considerable technical and political issues for traditional models of evaluation (Edelman 2000). Community programmes have long time frames, they take place in communities where many other programmes are running at the same time, often with the same goals. Even more challenging, community programmes are usually based around a philosophy of community autonomy (Shirley 1982). This presents interesting tensions for evaluation when you come to look at whether or not a programme has met its objectives. Should the evaluation assess achievement of a set of objectives proscribed by the funder or a set of objectives set by the community itself, or both? There are models and approaches that can be used in the evaluation of such programmes (Duignan and Casswell 1989; Duignan and Casswell 1992; Duignan, Casswell et al. 1993; Moewaka Barnes 2000). These models and approaches need to be further refined as part of the essential tool kit building social policy evaluation of real world programmes. 

 

Not privileging any one meta-approach to evaluation

 

Meta-approaches to evaluation are evaluation styles that endorse a particular solution to the philosophy of science questions lying behind evaluation – in particular stakeholders interest in the truth-status of claims that are made in an evaluation.  Goal-free evaluation (Scriven 1972) and empowerment evaluation (Fetterman, Kaftarian et al. 1996) are good examples of meta-approaches to evaluation that take different philosophy of science positions (Scriven and Kramer 1994). It is fine for evaluators to adopt one or other of these meta-positions in their professional work as evaluators. However, in building evaluation capability it is important that a more inclusive approach is taken to evaluation that does not privilege just one approach to evaluation. Of course, the Western evaluation approach itself can be seen as just one meta-approach to evaluation and we need to be aware that this is not universally accepted by stakeholders. Maori are actively involved in the process of developing evaluation models and approaches which may or may not have similar assumptions, methods, and techniques to evaluation as it is practiced in the Western tradition  (Watene-Heydon, Keefe-Ormsby et al. 1995; Moewaka Barnes 2000; Smith 2000; Moewaka Barnes In Press).

 

Sophisticated Model of the Evidential Base that Evaluation can Deliver

 

The last element in the evaluation model needed for social policy capacity building is a sophisticated model of the evidential base that evaluation is likely to be able to deliver. There is a tendency in social policy to start with a naïve expectation that evaluation may be able to deliver the type of ‘evidential map’ that is illustrated in Diagram 2.

 

 

Diagram 2 Evidential Map of Links between Social Policy Programmes or Policies and Outcomes

 

Diagram 2 shows evaluation providing evidence linking a series of social policy programmes or policies to a series of cross-sector social outcomes. Everyone would acknowledge that because of resource and technical constraints, evaluation cannot provide a totally comprehensive map of these links. However it is important to distinguish between holding the view that we can approach a comprehensive evidential map as in Diagram 2 or whether our expectations of evaluation should be much more like what is set out in Diagram 3.

 

 

Diagram 3: The Likely Extent of the Evidential Map Delivered Through Evaluation

 

In the authors’ view, Diagram 3 provides a much more realistic picture of what evaluation is likely to be able to deliver in the social policy area, even when a large amount of evaluation is being undertaken. We are unlikely to ever get anything like a full evidential map on which to base rational social policy. We will continue to be forced to make substantial decisions under uncertainty. Within the evidential map there will, of course, be some connections between programmes and outcomes that are easier to measure than others.  These relatively easy to evaluate programmes will tend to be ones which:

·        Operate at only the individual level rather than including organisational, community and policy level strategies.

·        Take place in only one locality rather than at national and local level.

·        Focus on single outcome variables that are already routinely collected, rather than multiple outcome variables.

·        Take place in institutionalised controlled settings.

·        Seek outcomes that can be measured in the relatively short timeframe.

 

For instance, a school-based programme that uses examination results as its outcome measures is a good example of where it is relatively easy to measure and attribute changes in outcomes to the effects of a programme. 

 

It is important that, as we increase the evaluation activity taking place in New Zealand, we are realistic about what can be provided in terms of the social policy evidential map. Secondly, we need to understand the implications for social policy decision making of certain outcome evaluation designs being easier to implement in some social policy areas than in others. We cannot afford to become too simplistic about automatic application of outcome evaluation results to determining priorities for funding of programmes and policies.  This is particularly important as we move building the information base for ‘evidence based’ practice in social policy (Wright 1999).  The fact that quasi-experimental outcome evaluations are possible in some social policy areas should not be taken as evidence that quasi-experimental outcome evaluation is similarly feasible in all policy areas where alternative designs such as case study designs may need to be used. The amount of experimental outcome evidence for different types of programmes and policies is a function of both the actual effectiveness of the programmes and the ease of undertaking experimental outcome evaluations on the type of programme under consideration.

 

Given the current interest in “joined up solutions” in the social policy area (Maharey 2000), many of the programmes and policies that are currently being proposed have characteristics that mean they are more difficult to evaluate. They tend to:

 

·        Use a range of strategies at the individual, community and policy level in an integrated programme.

·        Take place at both local and national level at the same time.

·        Be directed at multiple rather than single outcomes, some of which may be expensive to collect data on.

·        Take place in uncontrolled community rather than institutional settings.

·        Seek long term outcomes that will take years to come to fruition.

 

In these cases experimental outcome evaluation is much more difficult. This does not mean that we should not attempt to undertake evaluations of such programmes, but that the evaluation designs we will use will have to be different. These evaluation designs, such as case studies, will yield different types of data from the quasi-experimental designs.  A more sophisticated approach needs to be taken to evaluating such programmes, using a range of  types and methods of evaluation as discussed early in this paper. There will of course still be situations in which experimental or quasi-experimental outcome evaluation is possible and should be undertaken if it will answer a priority evaluation question for the sector.

 

The issue of how comprehensive an evidential map evaluation can provide becomes particularly critical when attempting an evidence-based approach to prioritising interventions to achieve cross-sector social policy objectives. The author’s experience during a recent review of strategic social policy for the Ministry of Social Policy and the State Services Commission indicated that this sort of prioritisation was, naturally enough, on the wish list of politicians and policy analysts alike (Duignan and Stephens 2001). However such exercises can never become routinely empirically based (at least for the foreseeable future). Diagram 4 illustrates this point by looking at what the evidential map may look like in a limited selection of cross-linked social programme areas. Looking first at the left hand side of this diagram, it may be possible to use outcome evaluation results to attribute the outcome of a reduction in offending to small group programmes run by the Department of Corrections. However it is also likely that Te Puni Kokiri funded capacity building programmes working at the community level (and hence harder to evaluate for attribution of outcomes) may be also be contributing to reduced offending. Equally a community action programme funded through the Health Vote that includes an element of reducing alcohol abuse may also reduce offending.  Another example is provided on the right hand side of the diagram where a reduction in road accidents can be directly attributed through evaluation to Land Transport Safety Authority activity. However, a community action programme from the Health Vote may also reduce road accidents due to its targeting of alcohol abuse (but again because it is a community based strategy it is likely to be more difficult to provide outcome evaluation evidence to directly attribute reduced road accidents at the same level of certainty as can be done for Land Transport Safety Authority activity).

 

 

 

Diagram 4. An Example of the Evidential Map for a Selection of Social Programmes

 

Evaluation Models for Evaluating Programmes for Maori

 

An additional element, which is essential to any evaluation model used in capacity building in the social policy area, is providing for the evaluation of programmes for Maori. Since evaluation is about placing value on policies and programmes, it raises enormous issues in New Zealand in the context of the Treaty of Waitangi. At the current time a number of priority government programmes are targeted at Maori and so evaluation of these programmes obviously needs to be undertaken. Various guidelines have been issued regarding research and Maori (Te Puni Kokiri 1999; Health Research Council n.d.). The most basic consideration is whether methods that are to be used in an evaluation are going to work with Maori. However, the issue of evaluating programmes for Maori is much wider than this.  There are considerations at the level of: the governance and control of an evaluation; its cultural integrity; stakeholder support; the evaluation personnel; the methods used in the evaluation and the dissemination and control of data from the evaluation. While there is considerable ongoing discussion of research methods from a Maori perspective (Smith 2000), specific models for the evaluation of Maori programmes are still emerging. In the longer term, a project is current underway, funded by Te Puni Kokiri, that is looking separately at government expectations of evaluation of Maori programmes and Maori perspectives on evaluation models, looking for commonalities and divergences in these perspectives. Out of this recommendations may emerge regarding support for current and future work which is going on to develop models that work for Maori.

 

Building a Culture of Evaluation Through Appropriate Evaluation Skills Training at all Levels Within the Social Policy Sector

 

 

The next step in building evaluation capability is to develop skills, systems and structures for evaluation activity at all levels within the social policy sector, government, community sector and Maori. The objective of skills development in evaluation for the sector is to both further sophistication about evaluation along the lines of the evaluation model discussed above and to teach appropriate specific evaluation skills to those who can use them in their day-to-day work. This can be done by developing appropriate training materials and by running training workshops.

 

The Alcohol and Public Health Research Unit and Whariki have developed a series of manuals on evaluation that reflect the evaluation model described above and have been widely distributed throughout the public and community health sector in New Zealand (Casswell and Duignan 1989; Duignan, Dehar et al. 1992; Turner, Dehar et al. 1992; Waa, Holibar et al. 1998). As time has gone on the response to these manuals has been evaluated and subsequent manuals have been amended on the basis of this feedback.

 

During the period of time that the resources have been available, the Unit and Whariki have carried out a series of training workshops for different audiences within the sector. The different types of training are:

 

 

 

All of these courses, apart from the managers’ courses, involve both discussion of evaluation models combined with hands on working with evaluation projects brought to the workshops by participants. This action learning approach ensures that participants go away with a feeling of mastery in at least some evaluation techniques, which further assists in promoting the idea that there are aspects of evaluation which can be done by people at all levels within a programme, organisation or sector. 

 

There are signs that this sort of approach is starting to be utilized in other parts of the social policy sector. A number of government agencies have been building their evaluation units and obtaining evaluation training for their staff. Under the auspices of the Australasian Evaluation Society there have been evaluation courses run in Wellington as part of this. The most systematic approach which has come to the attention of the author is the Department of Work and Income’s capacity building work within its own organisation to boost its evaluation capacity. Given the Department’s key role in social policy it will be interesting to see if capacity building approaches such as those developed by the Alcohol and Public Health Research Unit and those undertaken by the Department of Work and Income spread to yet other parts of the social policy sector. 

Organisational or sector level strategising to prioritise evaluation questions.

 

The last essential element for social policy sector evaluation capacity building is to foster sector level strategizing about what are priority evaluation questions for developing social policy in New Zealand. These discussions need to involve government, Maori and community Sector stakeholders.

 

In thinking about such strategic discussions it is important to distinguish between discussions about evaluation and performance measurement. Performance measurement is a routine activity which uses relatively cheap and easily collected measures to monitor whether a programme is delivered. Evaluation, on the other hand, is generally a more strategic, more expensive, more detailed and one-off activity in which particular programmes are selected for evaluation. This can include the usually more difficult to measure question of whether the programme is achieving its objectives. Both performance measurement and evaluation are essential. However when their purposes become confused and one starts to predominate, as some commentators believe has become the case in North America, this causes major problems (Perrin 1998).  One mistake is inappropriate attempts to make unrealistic “evaluation-type” evidential demands on a programme to ‘prove’ that it is achieving hard to measure outcomes as part of its routine performance measurement. 

 

The relationship between strategy, evaluation and performance measurement can be characterised as set out in Diagram 5.

 

 

 

Diagram 5: Relationship Between Strategy, Evaluation and Performance Measurement

 

Diagram 5 is based on the notion that decision making about outputs, performance management and evaluation should take place as a part of a strategic discussion between Ministers, central agencies, and agencies. These strategic discussions should involve Maori, government and community sector agencies in the roles of both policy player and provider. These discussions, which will take place in a number of forums, should:

 

·           Take into account all the evaluation findings that have been established up to that time point in time to determine what outputs/programmes are likely to achieve the outcomes being sought by the government.

·           Specify the outputs/programmes that need to be delivered by agencies in the next period.

·           Specify performance management processes to ensure that the agencies deliver the outputs/programmes they have agreed to deliver.

·           Determine the next round of priority evaluation questions that will better inform the next stage of the strategic conversation. 

 

The result of this sort of approach should be to move the emphasis away from just thinking about evaluation on the basis of the “technical” programme evaluation characteristics of a particular evaluation and towards the ability of evaluation activity to contribute to the ongoing strategic social policy debate. A less technically sophisticated evaluation that can add more to the strategic conversation is of more value than a technically sophisticated evaluation the results of which may, however, distract from ensuring the most strategic use of scarce public sector resources.

 

One useful way of looking at evaluation from this different perspective is to see the spending on evaluation as part of one big Research and Development fund available for social policy development. This highlights the role of the government, central agencies, Maori and the community sector in defining evaluation questions, rather than seeing evaluation as something that is “done to” programmes or agencies. This also helps to move away from some of the undesirable aspects of the growth of an “evaluation industry” based around individual programmes and agencies which tend to stem from obligatory programme and agency evaluation regimes (Bushnell 1998).

 

Of course, it can be argued that already a lot of organisational and sector strategic considerations are factored into the evaluation requirements for an individual programme. Funders will indicate which programmes they want evaluated, the level of resources, and may indicate which evaluation questions they want answered. In addition, in reviews of the academic literature, and in priority setting processes within research funding bodies there will be prioritisation happening. In those sectors where there are ongoing research groups, involved in teaching, advising and undertaking a large number of evaluations, they will, in part, play a role through having a strategic view of a sector and which evaluation questions are the next priority. The Alcohol and Public Health Research Unit and the Injury Prevention Unit at the University of Auckland have played this role in regard to their respective sectors over a number of years. However, for the social policy sector this process is still too ad hoc and there is often a disjunction between priority setting, in those instances when it is taking place, and what actually happens on the ground in regard to the evaluation of the many programmes that are subject to evaluation.

 

Exactly how to facilitate social policy sector evaluation prioritisation, outside of the contribution made by dedicated research units, is a difficult question. Sectors dealing with social issues tend to be made up of a diverse range of public and private groups funding a diverse range of programmes. There are some innovative evaluation priority setting exercises going on in New Zealand at the moment in the labour and employment program area (McKegg, 2000).

 

Conclusion

 

This paper has looked at developing evaluation capacity in the social policy sector in New Zealand. The increased interest in evaluation has been greeted enthusiastically by evaluators who have been working in evaluation in New Zealand over the last few decades.  It is particularly welcome after the lack of adequate attention to outcomes which occurred following implementation of the outputs/outcomes dichotomy introduced at the time of the public sector reforms in the 1980s. If we are to make the most of the current interest in evaluation, we need to focus on building evaluation capacity at all levels and in all parts of the social policy sector. This will require a rather different approach from only focusing on developing large scale technically sophisticated evaluations of social policy interventions. There is a place for such evaluations when they are targeted at answering strategic evaluation questions for the sector, adequate resources can be brought to bear on them, and they are feasible in terms of the programme or policy being evaluated. However these types of evaluations need to be embedded within a social policy sector in which everybody sees evaluation as their responsibility. Everybody within the sector also needs to be equipped with the skills to undertake appropriate types of evaluation in their day-to-day work. What we need now is resources for training and awareness raising in regard to appropriate evaluation models and methods and hard thinking about the difficult task of how we can fostering richer and more comprehensive discussions to identify the priority evaluation questions right across the social policy sector involving both government and the community sector and Maori stakeholders.

Acknowledgements

 

The author would like to acknowledge all of those who have worked at the Alcohol & Public Health Research Unit (APHRU) and the Maori research unit Whariki which works in partnership with APHRU. He would also like to acknowledge Jane McCann and Carolyn Lane of the organizational development company Navigate where he worked while undertaking additional social sector evaluation work.

Author Note

 

Dr Paul Duignan is a Senior Lecturer at the Alcohol and Public Health Research Unit, University of Auckland (www.aphru.ac.nz, p.Duignan@auckland.ac.nz) where he teaches two postgraduate papers in programme evaluation. He also works halftime as an evaluation strategist and facilitator consulting on evaluation design in the public sector (www.parkerduignan.com, paul@parkerduignan.com).

Bibliography

 

 

Bushnell, P. (1998). “Does Evaluation of Policies Matter?” Evaluation 4(3): 363-371.

Casswell, S. (1999). Evaluation research. Social Science Research in New Zealand: Many Paths to Understanding. C. Davidson and M. Tolich. Auckland, Longman.

Casswell, S. and P. Duignan (1989). Evaluating health promotion: A guide for health promoters and health managers. Auckland, Department of Community Health, School of Medicine, University of Auckland.

Casswell, S. and P. Duignan (1989). Evaluating health promotion: A guide for health promoters and health managers. Auckland, Department of Community Health, School of Medicine, University of Auckland.

Chelimsky, E. and W. R. Shadish, Eds. (1997). Evaluation for the 21st century: a handbook. Thousand Oaks, California, Sage.

Compton, D., M. Baizerman, et al. (2001). “Developing evaluation capacity while improving evaluation training in public health: the American Cancer Society's Collaborative Evaluation Fellows Project.” Evaluation and Program Planning 24(1): 33-40.

Controller and Auditor-General (2000). First Report for 2000: Health, School Board of Trusteed, Impact Evaluation. Wellington, New Zealand Office of the Controller and Auditor-General.

Cook, T. and D. T. Campbell (1979). Quasi-experimentation: design and analysis issues for field settings. Boston, Houghton Mifflin Company.

Dehar, M., P. Duignan, et al. (1993). “Formative and process evalaution of health promotion and disease prevention programmes.” Evaluation Review 17: 204-220.

Duignan, P. (1990). Evaluating health promotion: An integrated framework. Health Promotion Research Methods: Expanding the Repertoire Conference, Toronto, Canada.

Duignan, P. (1997). Evaluating health promotion: the Strategic Evaluation Framework. Psychology. Hamilton, University of Waikato: 328.

Duignan, P. (1999). Summary of Improving Public Sector Policy Through Quality Evaluation Conference, Wellington, New Zealand, 25-28 May 1999. Improving Public Sector Policy Through Quality Evaluation, 25-28 May 1999, Wellington, http://www.navigate.co.nz/discuss.htm#Evaluation Articles.

Duignan, P. and S. Casswell (1989). “Evaluating community development programs for health promotion: Problems illustrated by a New Zealand example.” Community Health Studies XIII(1): 74-81.

Duignan, P. and S. Casswell (1990). Appropriate evaluation methodology for health promotion. American Evaluation Society Annual Conference, Washington.

Duignan, P. and S. Casswell (1992). “Community alcohol action programme evaluation in New Zealand.” Journal of Drug Issues 22: 757-771.

Duignan, P., S. Casswell, et al. (1992). Promoting change in health promotion practice: A framework for the evalaution of health promotion. Psychology and social change. D. Thomas and A. Veno. Palmerston North, The Dunmore Press.

Duignan, P., S. Casswell, et al. (1993). Evaluating community projects: Conceptual and methodological issues illustrated from the Community Action Project and the Liquor Licensing Project in New Zealand. Experiences with Community Action Projects: New Research in the Prevention of Alcohol and Other Drug Problems (CSAP Prevention Monograph 14). T. K. Greenfield and R. Zimmerman. Rockville, MD, U.S. Department of Health and Human Services.

Duignan, P., M. Dehar, et al. (1992). Planning evaluation of health promotion programmes: A framework for decision making. Auckland, Alcohol and Public Health Research Unit, School of Medicine, University of Auckland.

Duignan, P. and B. Stephens (2001). Review of the Ministry of Social Policy's Strategic Social Policy Capability. Wellington, State Services Commission/Ministry of Social Policy.

Edelman, I. (2000). “Evaluation and community-based initiatives.” Social Policy 31(2): 13-23.

Fetterman, D., S. Kaftarian, et al. (1996). Empowerment evaluation: knowledge and tools for self-assessment and accountability. Thousand Oaks, CA, Sage.

Guba, E. G. and Y. S. Lincoln (1989). Fourth generation evaluation. Newbury Park, California, Sage.

Health Research Council (n.d.). Guidelines for Researchers on Health Research Involving Maori. http://www.hrc.govt.nz/Maoguide.htm#Introduction.

Maharey, S. (2000). Soical Directions Briefing. http://www.executive.govt.nz/minister/maharey/directions/index.html.

Mathison, S. (1991). “Role conflict for internal evaluation.” Evaluation and Program Planning 14: 173-179.

McClintock, C. (1986). “Towards a theory of formative program evaluation.” Evaluation Studies Review Annual 11: 205-223.

McKegg, K, (2000). Personal Communication.

Minnett, A. M. (1999). “Internal evaluation in a self-reflective organization: one nonprofit agency's model.” Evaluation and Program Planning 22(3): 353-362.

Moewaka Barnes, H. (2000). “Collaboration in community action, a successful partnership between indigenous communities and researchers.” Health Promotion International 15: 17-25.

Moewaka Barnes, H. (In Press). “Kaupapa Maori: Explaining the ordinary.” Pacific Health Dialog.

Patton, M. Q. (1986). Utilization focused evaluation. Newbury Park, Sage.

Perrin, B. (1998). “Effective Use and Misuse of Performance Measurement.” The American Journal of Evaluation 19(3): 367-379.

Rossi, P. H. and H. E. Freeman (1989). Evaluation: A systematic approach. Beverly Hills, Sage.

Scheirer, M. A. (1994). Designing and using process evaluation. Handbook of Practical Program Evaluation. J. Wholey, H. Hatry and K. Newcomer. San Francisco, Jossey-Bass: 40-68.

Schick, A. (1996). The  Spirit of Reform: Managing New Zealand's State Sector in a Time of Change State. Wellington, State Services Commission.

Scriven, M. (1972). “Pros and cons about goal-free evaluation.” Evaluation Comment: The Journal of Educational Evaluation 3(4): 1-7.

Scriven, M. (1991). Evaluation Thesaurus. Newbury Park, Sage.

Scriven, M. and J. Kramer (1994). “Risks, rights and responsibilities in evalaution.” Evaluation Journal of Australasia 9(2): 3-16.

Shirley, I. (1982). Development tracks: The theory and practice of community development. Dunmore Press. P. North.

Smith, L. T. (2000). Decolonising Methodology: Research and Indigenous Peoples. London, Zed Books.

State Services Commission (1999). Essential Ingredients: Improving the Quality of Policy Advice. Wellington, State Services Commission.

Te Puni Kokiri (1999). Evaluation for Maori: Guidelines for Government Agencies. Wellington, Te Puni Kokiri.

Tessmer, M. (1993). Planning and conducting formative evaluations. London, Kogan Page.

Turner, A., M. Dehar, et al. (1992). Doing evaluation: A manual for health promotion workers. Auckland, Alcohol and Public Health Research Unit, University of Auckland.

Waa, A., F. Holibar, et al. (1998). Programme evaluation: an introductory guide fo health promotion. Auckland, Alcohol and Public Health Research Unit/Whariki Runanga, Wananga, Hauora me Te Paekaka, University of Auckland.

Watene-Heydon, N., V. Keefe-Ormsby, et al. (1995). The issue of research by indigenous peoples. Australasian Evaluation Society Conference, Sydney.

Wright, J. C. (1999). Working Paper No. 8: Strategic social policy advice: Improving the information base, State Services Commission.

 

 

 

 

V4 11-9-01 www.parkerduignan.com/documents/100.htm