Systematic Outcomes Analysis identifies a set of issues in the management of monitoring and evaluation as follows:
Issue 1: Consultation with stakeholders regarding monitoring and evaluation planning, implementation and review. The consultation processes with internal and external stakeholders regarding monitoring and evaluation planning, implementation and review, needs to be detailed in any monitoring and evaluation planning.
Issue 2: Evaluation management structure. The structure for governance, management and undertaking evaluation activities needs to be clearly determined in monitoring and evaluation planning. In small evaluations this will all be managed within existing governance and management structures. In larger evaluations, specific governance structures may be set up (e.g. Steering groups), technical input may be sought (e.g. Technical Advisory Groups), structures to manage internal evaluation staff will be established, and systems for contracting with external evaluators will be established.
Issue 3: Internal versus external evaluators. There are pros and cons with using internal versus external evaluators and these need to be considered in evaluation planning. These are as follows:
• Integration with strategic decision-making. Likely to be more with internal rather than external evaluators.
• Independent judgement. Likely to be less with internal rather than external evaluators.
• Cost. Likely to be less with internal rather than external evaluators.
• Range of evaluation skills. Likely to be less with internal rather than external evaluators.
• Institutional knowledge retention. Likely to be more with internal rather than external evaluators.
• Getting the evaluation done. Likely to be less prioritization of the evaluation by internal staff distracted by other organizational priorities than in the case of external evaluators.
• Drift in the evaluation questions being answered. Likely to be less 'evaluation question drift' (where the evaluation does not end up answering the questions stakeholders thought it going to answer) with internal rather than external evaluators. Once exception to this is if political factors within the organization change the direction of the evaluation and in this case, internal evaluators are more likely to be influenced than external evaluators.
Issue 4: Knowledge management. Evaluations create large amounts of information which needs to be managed both at the time the evaluation is being run and so that the findings and lessons learnt from the evaluation can continue to inform strategic decision making in the future. This requires careful attention to knowledge management systems within and across organizations.
Issue 5: Risk management. There are a number of risks which need to be managed in monitoring and evaluation. The general risks faced by any monitoring and evaluation are set out below with the way in which these are managed when using Systematic Outcomes Analysis.
• Not asking and answering the right monitoring and evaluation questions. Systematic Outcomes Analysis, by linking evaluation questions back onto the outcomes model and by going through the set of seven areas of evaluation focus increases confidence that all important evaluation questions have been identified. Having your Systematic Outcomes Analysis peer reviewed increases the chances that you are considering the right questions.
• Lack of stakeholder confidence in the independence of monitoring and evaluation. The more transparent an evaluation is the more stakeholders are able to decide if they have confidence in it. Systematic Outcomes Analysis provides a fully transparent view of an evaluation. Appropriate consultation with stakeholders (1 above); the right evaluation management structure (2 above); and the right selection of internal versus external evaluators (3 above) all decrease the risk of a lack of stakeholder confidence.
• Getting evaluators with the right skills to undertake the evaluation. This depends on the actual availability of evaluators with the right skills to undertake the evaluation plus having the funding available to employ or contract them. Systematic Outcomes Analysis will help you clarify the exact evaluation questions you want answered (e.g. high level outcome evaluation questions versus other evaluation questions) and this is likely to help you when assessing whether potential evaluators have the skills needed to answer particular questions. Involving an additional independent expert evaluator on selection panels and advisory groups can mean that more informed decisions are made about whether evaluators being employed or contracted have the right skills.
• Drift in the evaluation questions being answered away from those stakeholders think are being answered. Systematic Outcomes Analysis makes sure that all stakeholders are clear about exactly which evaluation questions are being answered. Equally importantly (and often neglected in evaluation planning) it also identifies which evaluation questions are not being answered and the reasons why. Visually mapping all evaluation questions back onto the outcome model reduces much stakeholder and evaluator confusion about what is and what is not being answered.
• Lack of ongoing control of externally contracted evaluations due to contracting organization staff turnover. For evaluations which take a number of years this can be a major problem. Systematic Outcomes Analysis reduces this risk by providing a transparent evaluation plan which should be progressively updated throughout the evaluation so that any new staff can quickly understand what the the details of what the evaluation is trying to do and how it is going about it.
• Lack of integration of monitoring and evaluation. These two aspects are often not well integrated. Systematic Outcomes Analysis fully integrates them.
• Disconnect between evaluation findings and ongoing strategic planning. Systematic Outcomes Analysis can be used to fully integrates evaluation with strategic planning if the same outcomes model is used for both evaluation planning and strategic prioritization.
Issue 6: Evaluation costing. The potential cost of answering evaluation questions needs to be identified in monitoring and evaluation plans if decision makers are to make rational monitoring and evaluation resource allocation decisions. In addition, evaluation cost estimates are needed for budgeting once evaluation projects commence.
[V1.1]