00:00:00 Introduction and poll results

Eliza: Before we start, Victoria and I thought it would be a great idea to get some insights from our audience to make this session more interactive. We have three questions for you: whether you use the OECD evaluation criteria in your work, whether you use quantitative data to support your evaluation deployment, and if you are seeking ICT4D tools to assist with evaluation.

Looking at the results, we see a nice majority actually use the OECD criteria in their work currently. Regarding the utilization of quantitative information, there is a bit more variation; the majority use it extensively or to some extent. Less than 50% are actively searching for ICT tools, though a significant percentage, about 37%, are considering ICT tools as an option.

For those who use the OECD criteria, we hope to provide some food for thought with today's presentation. For those who don't, we aim to provide a good starting point. For those considering ICT tools, especially for quantitative information, we will see how ICT4D can support the impact evaluation process.

00:02:54 Understanding impact evaluation

Victoria: Today, the outline of our discussion involves providing guidance on impact evaluation, key considerations, and what sets the evaluation of humanitarian action apart. We will then move on to working with the OECD-DAC criteria, introducing or reintroducing them as a refresher. We will go through the six different criteria and discuss how we can use ICT tools to facilitate evaluation before opening the floor for questions.

Eliza: The starting point of the OECD-DAC criteria is impact evaluation. It started back in 1991 when the Development Assistance Committee (DAC) of the OECD set out broad principles for evaluation. Initially, these started as five criteria used for development initiatives. Subsequently, they were refined into a set of six criteria and adapted to the needs and context of emergencies and humanitarian settings.

Over the past years, we realized these have been applied widely, which has important advantages. It allows for easier evaluation synthesis, standardization, and better comparison. It serves as a common vocabulary, making it easier to identify common weaknesses and strengths across different projects and countries.

Impact evaluation is the methodological assessment of changes in outcomes that can be attributed to a specific intervention, such as a program, project, or policy. Its significance lies in using empirical evidence to determine whether an intervention has achieved its intended outcomes. This leads to insights on effectiveness and efficiency. Evaluation also supports the identification of positive and negative, primary and secondary, and long-term effects caused by a specific intervention.

Evaluation is an assessment supposed to be systematic and impartial to draw lessons learned, improve policies and practices, and improve accountability towards donors, stakeholders, and beneficiaries. It is often undertaken by an external evaluator or a mix of external and internal staff to maintain impartiality while leveraging internal insights.

The role of evaluation in humanitarian action is crucial to assess the effectiveness of our actions and be accountable to beneficiaries. However, the context of emergencies has particularities. These are often implemented in periods of severe disruption and fast-changing environments. Challenges include accessing key informants, managing risks regarding personal information, and setting up MEAL systems quickly. There is an increased need for flexibility, lighter methods, and adherence to humanitarian principles like "do no harm."

00:15:42 Principles of impact evaluation

Eliza: It is useful to keep the principles of impact evaluation in mind. First, it should be relevant and appropriate to the specific country and period. We need to be credible, using methodologies we trust to yield reliable results. We need to be realistic and feasible, considering the context, resources, time, and staff capacity available.

We must also consider the ethical component and humanitarian principles. We need to ensure transparency and participation, maintaining openness about our methodologies and findings. Finally, we need to produce something useful that allows us to learn and be accountable to our stakeholders.

00:18:12 The purpose and principles of OECD-DAC criteria

Victoria: The OECD stands for the Organization for Economic Co-operation and Development, and DAC stands for the Development Assistance Committee. The evaluation criteria serve the purpose of assessing the value, worth, and significance of an intervention. They provide different lenses through which the intervention can be evaluated, offering a comprehensive understanding of the implementation process and results.

The criteria have a normative role, establishing standards such as relevance, coherence, and efficiency. They also serve the purpose of accountability by providing information to donors and the public, and they support learning through findings and lessons. Beyond evaluation, they can be used to monitor progress, manage results, and design future interventions.

There are two key principles to bear in mind. Principle one emphasizes understanding the criteria within a broader context. They should be applied thoughtfully and contextualized to the specific intervention and stakeholders. Principle two highlights that the application of the criteria should align with the purpose of the evaluation. It should not be a mechanical process; the criteria should be tailored to meet the needs of the stakeholders and the specific evaluation context.

00:23:25 Criterion 1: Effectiveness

Victoria: The first criterion is Effectiveness. Here we ask: Is the intervention achieving its objectives? Effectiveness refers to the extent to which an intervention has achieved, or is expected to achieve, its objectives and desired results. This involves analyzing the relative importance of the objectives and results.

A very important aspect of effectiveness is timeliness—whether the intervention was implemented within the appropriate timeframe and addressed needs in a timely manner. With timeliness comes preparedness. In evaluating effectiveness, we analyze the extent to which the stated objectives have been met.

To fully understand outcomes, it is important to identify the reasons behind success or failure. Utilizing a checklist of cross-cutting themes can help examine aspects like stakeholder participation. Furthermore, utilizing the Logical Framework Approach (LFA) simplifies the evaluation of effectiveness by analyzing specific objective statements. Finally, evaluation should investigate who utilizes and benefits from the provided resources, ideally gathering data segmented by gender, socioeconomic status, and ethnicity.

00:29:06 Criterion 2: Impact

Victoria: The next criterion is Impact. We ask: What difference does the intervention make? This assesses the extent to which the intervention has generated significant positive or negative effects. These effects can be intended or unintended, but the focus is on the broader consequences on the target group or the larger context.

While effectiveness examines whether outputs and objectives were met, impact examines the long-term consequences and wider social change. The significance of impact evaluation depends on timing and resources. Assessing impact immediately after a project closes may not be pertinent because socioeconomic changes take time to manifest. Impact evaluation often requires a longitudinal approach.

Attribution is a key challenge. Pinpointing the direct effect of an intervention becomes complex over time due to other influences. Interviewing control groups can help address this. When evaluating impact, it is important to link it to the goal and purpose section of your logical framework.

00:33:18 Criterion 3: Efficiency

Victoria: For Efficiency, we ask: How well are resources being used? This measures the extent to which an intervention delivers results in an economic and timely way. Efficiency is measured by comparing outputs to inputs. It involves assessing whether the intervention used the most efficient process to achieve desired outcomes.

Cost-effectiveness looks at whether different outputs could have been produced to have a greater impact. Considerations include political factors, such as government agendas or donor preferences, which can impact efficiency. The origin of inputs is also important; for example, sourcing supplies locally versus importing them can significantly affect efficiency. Financial aspects, including total costs by sector and administrative costs, play a crucial role.

00:36:20 Criterion 4: Relevance

Victoria: For Relevance, we ask: Is the intervention doing the right thing? This assesses the extent to which the intervention's objectives and design respond to beneficiaries' needs, global needs, and country priorities. It involves considering whether the design is sensitive to the context.

Relevance and appropriateness go hand in hand. Relevance is suitable for examining broader aspects like alignment with local needs and donor policies. Appropriateness is more suited to assessing specific inputs and actions. For example, in a health intervention reducing maternal mortality, relevance assesses if the objectives align with the priorities of the beneficiaries.

Understanding the local context and conducting a thorough needs assessment are essential. Evaluators need to examine how well the planning and design took into account local factors. Cultural suitability is also crucial; for instance, providing shelters with private spaces for women and girls in certain cultural contexts.

00:39:57 Criterion 5: Coherence

Victoria: For Coherence, we assess: How well does the intervention fit? This looks at the compatibility of the intervention with other interventions in the country, sector, or institution.

Internal coherence focuses on synergies and interlinkages between the intervention and other interventions carried out by the same institution or government. External coherence considers the consistency of the intervention with the interventions of other actors in the same context. It involves assessing complementarity, harmonization, and coordination to avoid duplication and maximize impact.

00:41:30 Criterion 6: Sustainability

Victoria: Finally, Sustainability. We ask: Will the benefits last? This assesses the extent to which the net benefits of an intervention continue or are likely to continue. It involves examining the financial, economic, social, environmental, and institutional capacities of the systems required to sustain the benefits over time.

00:43:18 Case study and using ICT tools

Eliza: To illustrate working with these criteria, we can look at a case study of emergency initiatives for vulnerable people in Iraq between 2017 and 2020. The evaluation methodology included a desk analysis of project documents and reports, combined with physical inspections and primary data collection through key informant interviews and focus group discussions.

Regarding Relevance, the assessment found that the agency had not systematically used all available means to participate in assessment activities, which impacted the equity of the project. regarding Coherence, they identified a disconnection between the multi-annual development strategy and the emergency response. For Effectiveness, the desk review showed that while monitoring systems were in place, they were inconsistent across partners, leading to decreased effectiveness.

This case makes a strong argument for using ICT4D technology. ICT tools act as a data repository, facilitate information sharing with stakeholders (including external evaluators), and keep data collection for implementation and evaluation in the same database. This streamlines the approach and decreases duplication of efforts.

Using ActivityInfo as an example, we can structure a database into three categories:

Having everything in one system allows you to link project participants to specific interviews and assign codes during analysis, facilitating the evaluation process.

00:55:49 Questions and answers

Victoria: We have several questions. One asks if we recommend using all DAC criteria when preparing a Terms of Reference (ToR). If the ToR is for an evaluation process, you should prepare it in line with the DAC criteria if that is the framework you have selected. You need to take into account the indicators you want to measure and the criteria against which you want to measure them.

Eliza: Another question asks about the difference between Relevance and Coherence.

Victoria: In plain English, Relevance asks "Is the intervention doing the right things?" For example, if a community needs water and you provide a policy workshop, that is not relevant. Coherence asks "How well does the intervention fit?" It looks at how your intervention fits into the internal and external context. If the government is working on renewable energy and you are working on fossil fuels, that lacks coherence with the broader context.

Eliza: A participant asks if we can justify omitting the Impact criterion if resources are not available.

Victoria: Yes, if you do not have the resources to evaluate impact, it is impossible to do so. You can apply some criteria at one stage and others later. However, you must be transparent about the limitations. If there are no resources to conduct a longitudinal study required for impact, you cannot include it.

Eliza: There is a question regarding Timeliness: does it relate to efficiency or effectiveness?

Victoria: Timeliness is important for many criteria. For effectiveness, if an intervention comes ten months after a disaster, the timing is off. For efficiency, it is about the process—did you use resources in a timely way to achieve the goal? Efficiency is more about the process, while effectiveness is about whether the objective was met.

Eliza: A user asks if there are standard questions for each criterion and if one can add to them.

Victoria: While the criteria provide a blueprint, they can be adapted. You can add questions to elicit what you want to find. It is important to bear in mind the possibility for adaptability to measure the worth of things not directly covered by the standard criteria.

Eliza: How do we incorporate cultural responsiveness?

Victoria: Relevance is a key criterion where you can attach cultural dynamics. If there are cultural elements that influence how an intervention is received, you should find a way to include them in the project design so they can be evaluated, often under relevance or appropriateness.

Eliza: A question regarding Attribution: How can an individual organization assess its contribution when multiple partners are working in the same location?

Eliza: This is challenging. If beneficiaries cannot differentiate between services, it is best to conduct a joint evaluation. If that is not possible, you can use experimental or semi-experimental methods (like control groups) if resources allow. If not, you must simply be transparent about the limitations regarding attribution.

Victoria: Regarding the difference between Program Review and Program Evaluation: A program review is often internal, focusing on operations and processes (like After Action Reviews). Evaluation is generally more systematic, objective, and often involves external persons to determine if intended outcomes were achieved.

Eliza: Finally, regarding Outputs vs. Outcomes: Outputs are lower-level results, closer to activities (what we deliver). Outcomes are the responses or changes in the human beings receiving those actions. We need to assess both: what we offer (output) and whether that led to the intended change (outcome).