00:00:01 Introduction

Thank you, Faith, for the nice introduction. Hello everyone. Today we will dive deeper into the indicators. It's one of the most crucial topics in the MEAL system design process. In today's webinar, we aim to provide the basics for those that maybe are starting their careers or their project is in its early stages now, and a nice recap for those that actually have been practitioners for many years.

Just to brief you if you haven't watched the previous webinars, this is the third one. The first one focused on the Theory of Change and what does it mean, and how do we develop a Theory of Change and what are the basic elements. The second one focused on the Results Framework and Logical Framework. We saw the basic elements of the Results Framework and Logical Framework, what do they mean, and how we can develop those. This third one is focused on the indicators. Actually, indicators are considered an extra column or component of the Logical Framework, but also given their importance in the MEAL system design, we felt it necessary that they need a webinar of their own.

Today, we will do a quick recap of the Logical Framework elements. We will refresh on what is the Logical Framework, what are its elements, and how it is associated with monitoring and evaluation. Then we will dive deeper into indicators. We will understand why we use indicators, what is an indicator, what are its elements, and what are the characteristics of a good quantitative indicator. Last but not least, we will go through some steps on how to develop indicators and some best practices and tips around indicator development.

00:02:55 Recap of the logical framework

Starting a bit back from the Logical Framework, I just want to refresh your memory that actually the Logical Framework is part of what we call logic models, along with Theory of Change and Results Framework. Theory of Change aims to describe the long-term change, the pathways of change, and the underlying assumptions along with the supporting evidence. The Results Framework clarifies or illustrates the project hierarchy and the causal logic of the model. One of the main differences between Theory of Change and Results Framework is actually that the Results Framework aims to describe what is within our project's intervention jurisdiction, while the Theory of Change should take into consideration assumptions that may be outside of our sphere of control.

Then we move into the Logical Framework, which builds upon the Results Framework and includes the high-level indicators, means of verification, measurement methods, and assumptions that need to be in place so that causal logic actually works. A quick refresh is that the Logical Framework actually is a logic model but has some key features: indicators that we will analyze today, measurement methods that we will explore in the next webinar, and assumptions. We need it because it makes us a step closer to the MEAL plan. This is the basis of our MEAL plan, and that is why the indicators are considered the heart of the MEAL system and of evidence-based decision making.

It's a process. Along with all the other tools, Theory of Change and Results Framework, project teams develop a common understanding. This common understanding between the different teams that operate within a specific entity or organization—programming teams, Information Management teams, and Monitoring and Evaluation teams—allows them to use it during project implementation.

A quick reminder of what M, E, A, and L mean. We have Monitoring (M), which is the continual, systematic data collection of information about project progress, and Evaluation (E), which is considered the systematic assessment of the design, implementation, and results for an ongoing or completed project. Frequently, there is a confusion about when monitoring stops and evaluation begins. In reality, one of the ways that monitoring and evaluation differ is related to the questions that they ask and answer, and also the different frequency, purpose, timing, and data use. For example, monitoring focuses on tracking activities and outputs, making it more timely and ongoing. Evaluation happens on a higher level, as an objective assessment of the worth of the project. Monitoring serves better the fact that it enables us to perform real-time changes, while evaluation contributes more to the long-term learning of the organization.

Accountability is our commitment to respond and balance out the needs of different stakeholders. Projects embrace accountability by promoting transparent communication, sharing monitoring and evaluation information with relevant stakeholders, community partners, donors, and stakeholders. Learning is the culture that we have within a specific place and organization that enables us to reflect upon the data that we have, change, modify, and learn through the process. Project teams usually learn by encouraging the spirit of curiosity, embedding learning processes, and sharing information to promote adaptive management.

How do these correspond to our logical framework? Usually, monitoring happens more frequently and timely, so it better corresponds to the outputs and activities level and intermediate results. Evaluation more frequently goes to the strategic objective level and intermediate results as well. The reasons I mentioned are crucial because how timely we want the information and how we want to use this information enables us to choose the appropriate indicators for each level in order to be able to track our progress towards specific objectives.

00:11:16 Why are indicators important?

Indicators are important because they ensure evidence-based decision making. If we don't know what we count, what we measure, and how we measure, how can we actually use this information? They support our project's capacity to adapt timely. If we differentiate between the levels of output, intermediate results, and the objective, and we provide the relevant indicators to the lower level, we support our programming teams to use this information timely and see whether they need to change something in the specific activities. They support learning as we learn throughout the process from the adaptations that we need to perform. They also support accountability to key stakeholders because we share information with them regarding why we change something or why we keep the intervention as it is.

Of course, it's necessary to measure the indicator. It is also necessary to identify a target. A target is a specific result that we aim to achieve within a specific time period. For example, if we have an output level indicator like "number of refugees who participate in vocational training," we might have a monthly target of 100 people. Our target helps us identify to what extent this works well or not. If we aim to succeed in our target, this means that we do a good job with those trainings. If we are far from this target, this means that we need to revise either the activity or the target itself.

It's important also to identify a starting point. Sometimes we need to acknowledge that the moment we start the intervention, there's already something happening. The value of this indicator at the point in time right before our project starts is called a baseline. It is the value of this indicator that our project has before the starting point. For example, if our starting point was 10 and our target was 100, then what we did through the project intervention is 90 additional refugees if we managed to succeed.

00:14:53 What is an indicator?

First, indicators are qualitative or quantitative variables or factors. The main purpose is that they provide a reliable means to measure achievement. They enable us to track to what extent we can actually achieve the objective statements. They enable us to reflect on the changes and identify through the data use whether actually our intervention has worked well or hasn't worked very well. Indicators are a measure of progress through the project Pathway of Change—from activities, outputs, intermediate results, specific objectives to goal.

Regarding indicator elements, we have quantitative and qualitative. Quantitative indicators are numerical values. They have specific aspects: unit of measurement (number, percentage), subject of measurement (household, project participants), description of what is being measured, and disaggregation requirements (gender, age). Qualitative indicators aim to generate narrative information—text, words. They tend to explore and describe our opinions and attitudes towards a specific situation. For example, asking what challenges refugees face in order to access a specific labor market.

Most often, quantitative indicators are worded in a neutral manner. This means that they do not indicate the direction of change (increased or decreased), and neither do they have a target embedded in them. Sometimes, however, donor requirements mean that indicators may be written in a way that incorporates those elements. Quantitative indicators help us understand how much of something is happening. Qualitative indicators help us understand the why and the how of the process, and it's important especially in those circumstances where we are not sure what to expect.

00:21:51 Characteristics of a good indicator

The word is SMART, which means in practice: Specific, Measurable, Attainable, Relevant, and Time-bound.

Specific: We need to have a common understanding of what the indicator means. We need to be specific on the words that we include in this indicator, and we need to have them well-defined. When we have percentages, we need to be quite sure which is the numerator and which is the denominator.

Measurable: We need to be able to measure those indicators. Imagine that you choose an indicator that is hard to measure; most probably we cannot collect the indicator in a timely manner. We need to ask ourselves whether we can count or observe this indicator. We need to be pragmatic regarding the efforts of analyzing and collecting this indicator and whether the efforts correspond to the value of this indicator for our programming needs.

Attainable: We need to be sure that we can achieve this indicator. This component is pretty obvious regarding the target for this indicator. Can we reach the target? Or have we set up a target that is not achievable given the timeframe?

Relevant: It means that it corresponds to the objective statement. Each indicator is tailored to each objective statement of the logical framework or the results framework. We need to be sure that the indicator indeed helps us track the progress towards the specific objective.

Time-bound: We need to be sure that the frequency of data collection is appropriate given the frequency of those activities. Imagine the situation where we want to collect the information for an indicator on a weekly basis, but we haven't checked with the programming team whether the activities are on a weekly basis.

Putting the indicator in a SMART way is a reality check regarding whether we can measure the indicator and the quality dimension. When we have complex indicators, we always have the MEAL plan. The MEAL plan enables us to add those specifications. For example, if we have an indicator regarding "long-term job opportunities," the MEAL plan would have a column for the definition where we would say that long-term means a year.

00:27:09 Practical examples

Let's go to the example that has been used in the past webinars: the country of Homeland. This is a country that has received a great influx of refugees who want to build a future there. We are working as part of the MEAL team in the project proposal development stage.

Recapping the results framework:

Let's look at an indicator for the strategic objective level: "Percentage of refugees employed in long-term job opportunities."

Let's see another example for Intermediate Result 2 (increased skills). We could measure this in three different ways:

These three different indicators can lead to the same result approximately, but they differ in resources required for data collection. We might exclude the third option if we don't have the budget or people to collect data at two points in time, opting for a simpler solution like the second one.

00:34:17 Steps to develop indicators

Before diving into the steps, we identify the who, when, and how.

The process generally has four steps:

In an emergency, things are different. We want simple indicators that we can use frequently, often associated with the delivery of a service or items. We can use donor-provided indicators, industry-recognized standard indicators, or cluster indicators.

00:43:22 Best practices and tips

00:47:15 Q&A session

Could you please clarify what you mean by baseline? And is it necessary for all indicators to have a baseline?

No, it's not necessary for all indicators. Baseline means the value of an indicator before a project begins. For example, if we want to reduce malnutrition, the baseline is the existing malnutrition rate when we enter the country (e.g., 30%). If we reduce it to 25%, our contribution is the 5% reduction. Not all indicators have a baseline because sometimes we start from zero, such as with a totally new activity where no one else is operating.

Any tips on identifying which indicators are output or outcome level? Would you consider indicators on increased knowledge to be at the output or outcome level?

First, clarify the objective statements. If an indicator helps understand the move to the next level, I would put it on the lower level to get information more timely. At lower levels (output), we frequently have numbers (number of people, trainings). As we move up to outcomes/intermediate results, we frequently have percentages to identify increase or decrease. I try to put the less complicated indicators at the lower level to get data I can analyze frequently.

In the quantitative indicator example "percentage of refugees employed in Homeland," how important is it to add the duration the indicator is expected to be valid?

If the project runs for a specific year, we can add "by the end of the project implementation" in the wording. If the time component isn't in the wording, it can be added in the MEAL plan column that identifies when data is collected and analyzed.

Is it possible to associate the finance team when developing indicators, as we always need resources?

It is rare to include finance in the initial discussions as it's considered out of their area of expertise. I would give the lead to the programming team to identify useful information. If additional resources are needed, then communicate with finance to coordinate the budget. Finance would be included last in the process.

How do we find donor-mandated indicators for multi-country programs? There are instances that the specific mandated indicator is not useful for internal decision-making.

If it's a donor mandate, you need to include it in the proposal. If it's not useful for decision-making, the discussion becomes how to collect this information with the least effort. Start by defining the indicator to find the most straightforward collection solution to report back to the donor.

Nowadays, the industry is asking for more results instead of management. That said, would you agree that it is better to have less output indicators and more outcome indicators?

I would balance them out. You need evidence to back up the results. I wouldn't say add more in the next level because both levels have equal importance. You need to monitor the lower level to understand what you are doing. If you end up with too many indicators, filter them down by asking if they are useful to know what you need for project implementation.

How can we measure progress at endline if there's no baseline?

If there is no baseline, we mean by default it's zero. In this case, we measure our progress starting from zero. This is the most frequent case.

What is the difference between a process indicator and a proxy indicator?

Process indicators help us track our progress in terms of the process component of a defined objective. A proxy is a type of indicator used when we cannot measure an indicator directly. For example, if we cannot access employment data, we might use another type of information as a substitute (proxy) to confirm the percentages of people employed.

In addition to developing indicators for the objective statements in the LogFrame, would you recommend developing indicators for critical assumptions for learning purposes?

Yes, it could be the case. Frequently, monitoring critical assumptions requires qualitative additional information on the indicator level of the objective statements. You don't always need a separate indicator for assumptions; often, you need to keep in mind extra information to connect an indicator to the objective statement.

There are some domains in which the indicators should be defined with proxies, like peacebuilding or critical thinking. What would you suggest to an organization working on these issues when developing their indicators?

Proxies are one way to go if literature is available. Another way, especially for subjects like critical thinking or social-emotional skills, is community-defined indicators. It is worth the effort to conduct focus group discussions and identify what these concepts mean for the participants in that specific context.

Please clarify: the Theory of Change indicators and LFA indicators should be the same because the LFA usually is developed after the Theory of Change.

We do not separate indicators for the Theory of Change. The Logical Framework includes all the indicators we are going to track. When testing the Theory of Change, we use the indicators from the Logical Framework to see if indicators in Intermediate Results contribute to Strategic Objectives. You don't always need a new indicator for the Theory of Change; you use the Logical Framework indicators to test the logic.

How to determine realistic indicators (targets)?

One source is assessment data—what are the challenges and what is the timeframe? Discussions between programming and MEAL are crucial because programming teams know the context. The other reality check is whether we can do it within the project implementation period. An experienced program manager should be aware of how many people they can reach.

How do you justify not being able to achieve the indicator by the end of the project?

First, look at the deviation from the target. If monitoring goes well, you identify early that you will not reach the target and can communicate it timely. If monitoring hasn't been effective and you identify it at the very end, you need to reflect on the data to see what went wrong at each level. There is always a justification, provided the indicators chosen were meaningful.

What is the difference between output and performance-level indicators?

Performance indicators (KPIs) is often another name for indicators in general. If referring to the difference between output and intermediate results, it depends on the level. Outputs are usually numbers (delivered services), while intermediate results are about whether the person received the service, acquired the skill, or changed behavior.

In the definition of monitoring, we say that it's a continuous process. What does this mean in terms of how frequently we monitor the indicators?

In practice, we conduct monitoring frequently, not continuously (e.g., weekly, monthly). The frequency is project-specific. Data collection depends on when the activity is performed (e.g., monthly training = monthly collection). Then there is the question of when to process and visualize this information. You might gather the team monthly to reflect on data, or every three months if the project is going as expected.