1	Evaluating Health Promotion Today Context, Trade-offs, Being Strategic, REMLogic v1-2 Dr Paul Duignan Senior Research Fellow SHORE, Massey University Parker Duignan Ltd Wellington paul@parkerduignan.com www.parkerduignan.com www.strategicevaluation.info
2	"If using material from this..." If using material from this presentation please cite: Duignan, P. (2005). Evaluating Health Promotion Today - Context, Trade-offs, Being Strategic, REMLogic. Keynote Presentation to the Ministry of Health Public Health Intelligence Analytical Workshop, Wellington, 17 October 2005. (http://www.parkerduignan.com/pd/documents/127f.html)
3	Evaluating Health Promotion Today Demands for accountability Demands for evidence-based (informed) practice Demands for programme integration “joined-up” solutions Demands for community autonomy
4	Demands for accountability
5	Media Headlines “Who’s accountable?” (CNN Headline, 13 September 2005) “Brown puts blame on Louisiana officials: Former FEMA director defends Katrina response” (CNN Headline, 27 September 2005)
6	Demands for evidence-based (informed) practice
7
8	Demands for programme integration “joined-up” solutions
9
10	Demands for community autonomy
11
12	What is evaluation?
13
14
15	What is evaluation? Working out how to do things better
16	Evaluation terminology Evaluation approaches - utilisation-focused, empowerment, stakeholder, theory-based, strategic evaluation… Evaluation purposes - formative, process, outcome/impact. Evaluation methodology - intervention logic development, document analysis, observation, survey, focus groups, recording Evaluation designs - cases studies (quasi-)experiments. Source: http://www.parkerduignan.com/se/documents/104f.html
17	Evaluation purposes Formative evaluation - making sure programmes are optimised Process evaluation - describing what happened in the course and context of a programme Outcome/impact evaluation - assessing the positive and negative, intended and unintended results of a programme
18	Program life-cycle
19
20	So all we need to do is go forth and evaluate Right?
21	"Wrong" Wrong!
22	Problem with “go forth and evaluate” Only limited evaluation resources Bad evaluation worse than no evaluation Evaluation can have unintended consequences There are important trade-offs to be considered
23	Evaluation trade-offs
24	Firstly, evaluation difficulty
25
26	Individual level = easiest evaluation
27	Community level = harder evaluation Source: Paul Duignan
28
29
30	Potential value for money
31	Individual Level = Potentially lowest value for money health promotion
32	Community Level = Potentially more value for money health promotion
33
34	Examples where best value for money will require significant work at national level
35	Healthy nutrition
36
37	Road safety
38
39	Health inequalities
40
41
42	Trade-offs
43	Understanding the context of evaluation
44	Context Need to see any evaluation as part of an “outcomes system” Outcomes systems include strategic planning, indicator/outcomes monitoring, reporting and evaluation Looking at any element separately leads to distortions and duplication Outcomes Theory REMLogic* Framework identifies the building blocks. *Research, evaluation and monitoring intervention logic Outcomes methodology (for more information see resource list at end).
45
46
47	Conclusion: There is much to be done in evaluation Build evaluation capability at all levels Develop appropriate evaluation skills for people in different roles Appreciate the usefulness of formative, process and outcome evaluation in different situations Be very strategic about how we spend evaluation resources Understand the trade-off’s Better to do fewer evaluations of higher quality, than many low quality evaluations Get more coordinated about who is evaluating what Understand links between strategic planning, monitoring, reporting and outcome evaluation (REMLogic)
48	Further information Resources on Strategic Evaluation and Outcomes Theory available from www.strategicevaluation.info, including: Evaluation terminology: http://www.parkerduignan.com/se/documents/104f.html The REMLogic approach: 1) http://www.parkerduignan.com/se/documents/120pdff.html 2) http://www.parkerduignan.com/se/documents/126f.html Formative evaluation in government departments: http://www.parkerduignan.com/se/documents/121pdff.html Outcomes theory: http://www.parkerduignan.com/se/documents/122pdff.html Building evaluation capacity in a sector or organisation: http://www.msd.govt.nz/documents/publications/msd/journal/issue19/19-pages179-194.pdf Drawing an outcomes hierarchy (intervention logic): http://www.parkerduignan.com/se/documents/124pdff.html Monitoring community programmes: http://www.moh.govt.nz/moh.nsf/ea6005dc347e7bd44c2566a40079ae6f/48f14012c1e6f9facc256ccd00083dd8/$FILE/CPIF0203.pdf Monitoring centrally funded programmes implemented in different localities: http://www.parkerduignan.com/se/documents/107pdff.html