00:00:00 Introduction and recap

Welcome to the second webinar of our three-part series. The first session focused on how we get started with evaluation in humanitarian action. Today, we will move to the second one, which focuses on evaluation planning and design. We will mention some key takeaways from the first webinar to make the association with today's session, and the third webinar will focus on the implementation of evaluation.

In today's session, we will cover the design of an evaluation, including the different types of evaluation and how to frame the evaluation. Framing refers to the process of establishing a structure that will help us translate specific questions. We will discuss how to choose evaluation questions and the best practices around this choice, which is crucial for the quality of each evaluation. We will also look at the most commonly used evaluation criteria, specifically the OECD DAC criteria.

Then we will move forward with how we can plan for an evaluation and how we move from the Terms of Reference (ToR) to a specific action plan. This involves the concept of the inception phase, how we can gain knowledge in specific contexts, identify potential challenges, and the importance of the desk review. Finally, we will discuss best practices on how to manage and coordinate an evaluation and look at a real-case example of the evaluation of the Better Learning Programme implemented in Palestine.

To recap the previous webinar, the definition of humanitarian action impacts two main components: the decision to perform an evaluation and the scope of the evaluation. The nature of humanitarian action—including responding to crises, supporting disaster risk reduction, and recovery—is guided by specific principles such as humanity, impartiality, neutrality, and independence. Evaluation has two main purposes: accountability and learning. Learning is the process through which experience and reflection lead to changes in behavior, while accountability involves taking into consideration the views of different stakeholders, primarily those affected by the exercise of power.

Monitoring and evaluation are complementary tools. If an intervention has not been properly monitored from the beginning, it becomes challenging to evaluate. It may be appropriate to conduct an evaluation when piloting a new idea, when programs have unintended outcomes, or when there is a strategic interest from the agency or donors. A key challenge remains the use and uptake of evaluation findings.

00:06:50 Designing an evaluation: stakeholders and purpose

The first step in designing an evaluation is to identify the stakeholders who are interested in the evaluation and how they will use the findings. We need to identify the intended users—the people who are going to learn from the evaluation. Their involvement guides the choice of methods and tools and ensures that the evaluation contributes to improvement.

We need to ask: Who are the evaluation stakeholders? Who has a direct interest, such as donors? Who has an indirect interest? Crucially, who are the primary intended users? Stakeholders may have competing interests; for example, donors may be interested in effectiveness and efficiency, while program staff want to know what worked well during implementation, and senior managers may be interested in wider policy application.

The purpose of the evaluation determines how it is used. We can identify three categories:

00:11:22 Determining the type of evaluation

After identifying stakeholders and usage, we determine the type of evaluation. We must consider the scope (single project, programme, or sector), the level of results (process, impact, or outcome), the timing relative to implementation, and the actors involved (single-agency, joint, or system-wide).

Common types of evaluation based on scope include:

Based on the level of results, we have impact evaluation, which focuses on long-term effects (positive, negative, intended, or unintended), and process evaluation, which focuses on how inputs are converted into outputs.

Based on timing, we have real-time evaluation (ongoing during operation), mid-term evaluation, and ex-post evaluation (after the intervention is completed).

00:16:16 Framing the evaluation

Framing is the process of establishing a structure that helps translate evaluation questions. The starting point is often the programme logic—ranging from logical frameworks to Theory of Change—and normative standards for humanitarian action.

Project logic implies that planned action is based on a theory of how to achieve a desired result. Conceptual frameworks are logic models based on research illustrating key factors in an issue (e.g., Sustainable Livelihoods Framework). Logical frameworks show how inputs lead to long-term change. Theory of Change describes the central mechanism by which change occurs and is useful in complex environments to break results into causal linkages.

The second modality for framing is humanitarian standards. These guarantee that vital aspects of humanitarian action are not overlooked. Examples include the Core Humanitarian Standard on Quality and Accountability and the Minimum Standards for Education in Emergencies. These provide a reference point or threshold against which to evaluate performance.

00:20:54 Developing evaluation questions

Evaluation questions should be based on what the primary intended users need to know. They act as drivers of the evaluation, influencing the type of evaluation chosen, the design, the methods, the budget, and the sampling choices.

We have different types of questions:

We must avoid having too many questions or questions that are a poor fit for the scope. Reducing and filtering questions leads to a more focused evaluation, allowing the team to answer in more depth and providing more useful recommendations. We often "unpack" top-level questions (e.g., "Have we got the right strategy?") into underlying questions (e.g., "How can we improve our current strategy?").

00:26:59 Evaluation criteria

The OECD DAC criteria are useful tools to filter evaluation questions and determine if anything is missing. It is best practice to decide on questions first, then use criteria as a filter. The criteria were adapted for humanitarian action to include appropriateness, connectedness, coverage, coherence, coordination, and protection.

00:34:24 Evaluation planning and the inception phase

Once the design is set in the Terms of Reference (ToR), it must be translated into an action plan during the inception phase. This phase occurs prior to fieldwork and allows the team to develop a common understanding, define a specific plan and timing, and clarify ambiguities in the ToR. It is a critical quality control step.

Typical activities in the inception phase include initial interviews, desk reviews of documents (proposals, monitoring reports), and often a workshop to present the inception report and data collection tools. It is good practice to include an evaluation matrix in the inception report, which helps clarify questions, criteria, design, methods, and sampling in one place.

00:38:02 The importance of desk review

Desk reviews offer a cost-effective way to draw knowledge from previous evaluations and project monitoring documents. They help identify key issues and gaps in data. A desk review can take place as part of the inception phase to clarify tasks, as part of the actual fieldwork, or during the preparation of the ToR. It facilitates the evaluation by providing a base of evidence and context.

00:40:15 Managing and coordinating the evaluation

Managing an evaluation involves ensuring primary stakeholders remain engaged, often through a steering committee. The decision between internal or external evaluation depends on the purpose: external is better for accountability, while mixed teams are often better for learning.

When selecting external teams, criteria should include knowledge of the context, communication skills, fieldwork experience, and project management skills. Managing conflict is also important; steering groups acting in an advisory role can help minimize conflict arising from disparities between tasks and resources.

Timelines vary significantly based on context. Developing a ToR can take a week to a year. Fieldwork can range from a week for small evaluations to several months for large surveys. Dissemination usually occurs within a month of the report's publication.

00:43:05 Case study: NRC Better Learning Programme

To illustrate these concepts, we look at the evaluation of the Norwegian Refugee Council's (NRC) Better Learning Programme in Palestine. The program aimed to improve learning conditions for children exposed to war. The primary purpose of the evaluation was learning for NRC to guide future programs, as well as accountability.

They framed questions around impact (intended outcomes), relevance (response to needs), and appropriateness (sufficiency of resources). The methodology was based on OECD DAC criteria, complementing existing quantitative monitoring data with qualitative data collection, including stakeholder interviews, desk review, and "Most Significant Change" stories.

NRC established a steering committee to oversee the progress. The fieldwork took place over three weeks. The desk review included manuals, guidelines, proposals, and progress reports, which provided valuable information on the design and implementation stages.

00:48:15 Questions and answers

Can you elaborate on evaluation based on timing? Evaluation timing is closely associated with program implementation. For example, the ICRC conducted a real-time evaluation for a large investment to identify necessary changes early on. This differs from ex-post evaluations, which look back after a program has finished. The choice depends on how the findings will be used and project constraints.

What is the difference between designing and planning an evaluation? Design involves the decisions that dictate how you will perform the evaluation—determining the scope, questions, and methods. Planning involves the logistics of the actual fieldwork—how you will practically execute the design to answer the questions.

Do evaluation questions dictate the design and method? Yes, questions dictate the method. For example, if a question asks about beneficiary satisfaction, you likely need a quantitative survey. The method then influences the budget; a representative survey is more costly than key informant interviews.

Is there a recommended number of evaluation questions? There is no authoritative number as it varies by context. The important thing is that questions are filtered by the stakeholders involved. If a question will not result in findings used by the stakeholders, it should be removed.

How can an organization support smaller organizations to increase the usefulness of evaluations? The most important component is the planning stage, specifically setting up a strong monitoring system from the beginning. Without monitoring data, evaluation is difficult. Furthermore, findings must be communicated and acted upon through a clear action plan, rather than just filing the report.

Should the OECD DAC criteria be used in a specific order? No, the order does not matter. They serve as mental filters to ensure you haven't missed important aspects when formulating your questions.

How should a desk review be presented in the evaluation matrix? It depends on the purpose. If the desk review is a method for fieldwork, it is listed as a method answering specific questions. If used in the inception phase, it supports the rationale for the evaluation plan. It is rarely a standalone document but rather a source of evidence.

Can an organization carry out an internal impact evaluation and share it with a donor? Yes, as long as the reasons for using an internal team and the associated limitations are clearly listed and disclosed.

What if no evaluation plan is available and data is urgently needed in an emergency? If you cannot do a full evaluation, you may rely on secondary sources or proxy indicators. The first decision is whether an evaluation is feasible. If yes, but primary data collection is impossible, you rely on what is available via secondary data or proxy information.