00:00:00 Introduction and recap

Thank you, Faith, for the nice introduction. As mentioned, this is the last webinar of our webinar series. Just a recap for those that haven't watched the recordings or had the chance to join the previous webinars: We started with the Theory of Change. Then the next webinar focused on the Results Framework and Logical Framework, identifying some practices around their development. We moved forward with indicators development and the measurement methods that we showed in the previous webinar. Today is the last webinar of our series.

00:00:51 Agenda for the session

In a bit more detail, here is what we are going to see today. We are going to do a mini recap in terms of best practices of the indicators and the measurement methods that we showed in the previous webinars. This is because these are two crucial components that are strongly associated with the MEAL plan development.

Then we are going to move into the MEAL plan, which is also called a Performance Management Plan, or Monitoring and Evaluation plan, depending on the direction that each organization wants to take in terms of incorporating accountability and learning inside this plan. We are going to analyze what the MEAL plan is, why we use it, and what its main components are. Lastly, we are going to provide an example using ActivityInfo on how we can operationalize a MEAL plan with two different examples.

00:01:54 Recap on indicators and measurement methods

I have mentioned it in the previous webinar and I will mention it again because I think that it is crucial: indicators are the heart of any MEAL system. Why are they the heart? Because they ensure evidence-based decision making. They support adaptive management, learning, and accountability. Measurement methods are the enabling vehicle of the data collection process; they enable the initiation of the operationalization of the indicators.

Indicators are strongly associated with the purpose of the MEAL system, which is to enable the project's success. This can be done by collecting continuously and analyzing systematically information about project progress, which is known as monitoring. Here the indicators help us understand timely to what extent something is working well, or does not work well, via the systematic assessment of the design, implementation, and results of a completed or ongoing project, which is known as evaluation.

Indicators also support and enable the response to the needs of various stakeholders. Imagine that this is the vehicle by which we can provide information back to donors, beneficiaries, implementing partners, and other elements included in the project implementation. Then comes the learning, which is the active use of this information. It involves active reflection upon the indicator results and the progress made towards the achievement of our objectives to identify what is working and what doesn't work well in the project.

Some key messages to have in mind regarding indicators: Indicators are qualitative or quantitative. Quantitative indicators help us understand how much of something is happening, while qualitative indicators help us investigate the why and the how behind it. They help us investigate perception or motives regarding why people do specific actions. Qualitative indicators are critical to inform adaptive management actions, and frequently it's not a separate indicator, but an additional data point that helps provide context to the analysis and interpretation of quantitative indicators.

We also need to have in mind that they need to be SMART. This means the indicator can be defined (Specific); it is clear how the data can be collected and analyzed in a timely manner (Measurable); the nature of the change measured is achievable given our resources and the duration of the project (Achievable); the indicator is relevant to the different levels of the Results Framework (Relevant); and the indicator includes information on when our target is going to change in a specific timeframe (Time-bound).

00:06:22 Measurement methods

Regarding our measurement methods, they follow a similar logic to the indicators in terms of qualitative or quantitative. For quantitative indicators, we choose quantitative methods; for qualitative indicators, qualitative. Quantitative methods can be counted and subject to statistical analysis. They provide a very clear measurement, such as a number or percentage, and enable comparison between groups or across time.

On the other hand, qualitative methods capture participants' experiences; they explain how and why something is happening. This is implemented by stories, words, and pictures. Frequently, they trigger reflection and discussion, putting us in a position to analyze and identify themes. It is crucial to track the changes in participants' perceptions. This brings us to the use of mixed methods, which is using both quantitative and qualitative measurement methods. This is a great way of triangulation, where we can confirm a result about an indicator by using a qualitative method to get additional information.

We also need to have in mind that ICT tools are really powerful. Instead of using paper or Excel, it's worth considering ICT tools which make the operationalization of the measurements easier and give us access to timely information. We also need to consider our secondary sources. Before going out for any data collection, we need to check if we have this information from somewhere else and whether it is valid. Finally, we need to think about our context, the purpose of the data collection, and the resources—whether we have the right people, skills, and budget to implement the measurement methods.

00:09:38 MEAL plan development context

Where are we now with the MEAL plan development? The development of the Theory of Change, Results Framework, and Logical Framework has been implemented at the project proposal stage. That is when the programming team and the MEAL team decide the main logic of the program and how to measure progress.

The MEAL system starts building up before implementation. Let's say we are in the first week of implementing our activities. The programming team, MEAL team, and Information Management people sit in the same room again. They take whatever they had in the proposals and put it into a MEAL plan.

00:10:51 Defining the MEAL plan

The MEAL plan belongs in the broader category of MEAL planning tools. These tools enable the efficient operationalization of the MEAL system as a whole. They aim to respond to specific questions: how are we going to collect, analyze, interpret, use, and communicate our information? The purpose of the MEAL planning tools is to answer those questions and help project teams meet the information needs of all relevant stakeholders.

The primary planning tool is the MEAL plan: Monitoring and Evaluation plan, or Performance Management Plan. Other examples of those tools include the Indicator Performance Tracking Table (IPTT), the Feedback and Response Mechanism flowchart, stakeholders' communication plan, or learning plan.

00:12:15 Components of the MEAL plan

The MEAL plan aims broadly to answer a specific set of questions. These are:

The MEAL plan looks like a matrix with columns corresponding to this information. Frequently, the MEAL plan also includes key assumptions. It supports monitoring the assumptions during the project to check if they are still valid. If an assumption is invalid, you may need to redesign the project to address risks. Monitoring assumptions makes teams aware of changes early enough to make project adjustments timely.

00:15:00 Practical example: Homeland scenario

Let's see each component of the MEAL plan using the example we have used in the past webinars. Our specific country is called Homeland, and we have received a great influx of refugees. We are designing a project that addresses the challenges of integration of the target population. The programming team has made the proposal, got the approval, started activities, and now we are checking if we have everything to get started. The project manager has asked the MEAL team how he can get access to information timely.

We use the logical framework from the previous webinar. Let's focus on one indicator corresponding to intermediate result number one: "Vulnerable refugees are enabled to meet their urgent needs." The indicator is: "Percentage of refugees who agree or strongly agree with the statement: 'Referrals conducted by field team enabled me to access public medical services.'" We have a field team implementing referrals, and we are interested in the medical component. We decided to get this data via a survey.

00:17:06 Objective statements and indicators

The first component of our MEAL plan is the objective statements and the indicators coming from the logical framework. Note that the goal is usually not included in the MEAL plan as it represents long-term change not achieved during project activities.

Here we consider whether our indicators make sense and are well-defined. For our example, we ask: Is 'referrals' clearly defined? What do we mean by 'medical services'? Is psychosocial support included? What does 'access' mean? Is it when the field team books an appointment, or when the beneficiary attends the appointment? If access means attending the appointment, can we get information back? Can we find the beneficiary after they have completed the service?

00:19:04 Data collection and frequency

The second component is data collection. We have the measurement methods identified in the logical framework. We have our frequency, which depends on donor requirements, the needs of the programming team, when we expect to identify specific change, and our resources. We also identify who is responsible for the data collection and who our respondents are (e.g., head of household, female specifically). Identifying respondents involves sampling, which requires support from experienced MEAL practitioners.

In our example, the method is a survey. Due to donor requirements and programming team needs, we want this indicator bi-annually. It will be collected by our MEAL officers because we have the resources and want to reduce bias, as MEAL officers are external to the field team. We want to target all adults accessing the specific service.

00:21:19 Means of analysis

The third component is the means of analysis. Here we identify how we are going to analyze our information. Do we need a descriptive analysis (unique count, cumulative number, average)? Do we need qualitative analysis (coding)? We also identify our subgroups for comparison, such as gender (male/female) or family status. Comparison groups affect data collection; if we want to compare by family status, we must ensure that information is included in the data collection form.

In our example, we want a representative sample and we want to disaggregate by nationality.

00:23:12 Data use and flow

The data use component is a reality check on the utility of our indicator. We answer how our data will be used: Is it for donor requirements? What information does the program team need for decision-making? Do partners need information? In our example, we will use this information for bi-annual reflection and to submit our donor monitoring report.

After the MEAL plan, we have data flow maps. These visualize the flow of information from each data collection form to the corresponding report or use. It helps identify gaps in the system or places where we need to create a new form or report. It is possible that this mapping will generate ideas to update reporting and data collection processes. All tools used to support the MEAL system are living documents, so changes are expected.

00:25:20 ActivityInfo example 1: Secondary data collection

I will provide two examples using ActivityInfo to show how we can operationalize a MEAL plan.

Example 1: Secondary Data Collection. We are organization number one collaborating with three other organizations. We want to monitor partners using a common data collection system. We require them to report secondary information, such as the number of participants on a weekly basis who have completed vocational training.

In ActivityInfo, we create a form for secondary data. We add our selected indicators and additional information like calculation type, target, and responsibility. For the indicator "Number of participants for cash," the monitoring frequency is monthly, and it is cumulative. A partner (e.g., Partner 3) enters their unique beneficiaries for the period, disaggregated by gender (e.g., 32 females).

Since the program manager wants visual analysis and real-time information, we create a report. The report shows the total number of participants, disaggregation by gender (as listed in the MEAL plan), and a table showing participants per implementing organization. This is done by searching for the relevant form (partner reporting) and summing the reported numbers.

00:31:03 ActivityInfo example 2: Primary data collection

Example 2: Primary Data Collection. We are working in a scenario where our implementing partners are using our common data collection system. We have beneficiary registration, employment tracking, vocational training, post-monitoring, and a survey. We focus on the survey component because it provides the information for the indicator used in our example. All components are connected by beneficiary.

The survey is bi-annual, used for reflection and donor reporting. In ActivityInfo, we create a report or dashboard. ActivityInfo has a feature called dashboards which organizes information visually. We can create a two-page report: the first page provides an overview (number of respondents, gender/nationality disaggregation), and the second page inserts the specific information (how many people strongly agree, agree, etc., and the percentage). This report can be shared internally or externally.

00:34:22 Key messages

Some key messages to have in mind:

00:36:24 Q&A: MEAL plan vs. MEAL system

Question: How different is the MEAL plan and the MEAL system?

Answer: The MEAL system is like a big vehicle. The MEAL plan is like the engine of this vehicle; it is a tool that helps the operationalization of the whole system, along with other MEAL planning tools like the stakeholders' communication plan or the Feedback Complaint Response Mechanism. The MEAL plan is included in the MEAL system spectrum.

00:38:15 Q&A: MEAL plan vs. M&E framework

Question: What is the difference between a MEAL plan and an M&E framework? Or a MEAL plan and an M&E plan?

Answer: The difference between using the abbreviation MEAL or M&E depends on the incorporation of Accountability and Learning. Using "MEAL" puts an emphasis on the fact that I'm using it for accountability and learning, not just monitoring and evaluation. A Monitoring and Evaluation plan and a MEAL plan might look exactly the same, but the mentality differs. With MEAL, you share information back (accountability) and actively reflect on it (learning). This might lead to different approaches in indicators, frequency, or data use.

00:40:19 Q&A: Allocating responsibilities

Question: How do we allocate responsibilities of data collection to the colleagues if we don't include specific activities in the MEAL plan?

Answer: The data collection activity frequently corresponds to the measurements. If you say 'survey', it is a data collection activity. The person responsible for that specific activity corresponds to the activity illustrated in the MEAL plan. If you want more detail, you can use a MEAL calendar or a detailed implementation plan for MEAL activities, listing timelines and responsibilities more concretely.

00:42:21 Q&A: Pushing back on donor demands

Question: How can you push back against donor demands for top-down approaches to their LogFrame?

Answer: This depends on the indicator, context, and power dynamics. You can push back by focusing on the usefulness of the information. If data collection is hard to be effective or you cannot track respondents, use that to back up why you don't want a specific indicator. Also, consider how indicators are interpreted. Some donors provide strict definitions, while others provide broad ones or none at all. You can take advantage of the flexibility in definitions.

00:44:45 Q&A: Measurement methods vs. tools

Question: Do measurement methods equal measurement tools, and do these equal data collection tools?

Answer: The survey is a method for quantitative data. The tool for this is a questionnaire. So, the data collection tool and measurement tool are the same, but the method is the "how." For example, a Focus Group Discussion is the method, and the Focus Group Discussion guide is the tool. There is a difference between method and tool.

00:47:16 Q&A: Qualitative indicators in ActivityInfo

Question: Can you use ActivityInfo for qualitative indicators? Could you show an example of how the analysis report would look?

Answer: It is doable. You can use ActivityInfo for qualitative indicators using options like tags or multi-line text. However, analysis reports are more significant and common for the quantitative component.

00:48:22 Q&A: Means of verification for partners

Question: If working with partner organizations, what kind of Means of Verification (MoV) would you require? Is an annual report enough, or do you need attendance lists?

Answer: It depends on internal and donor requirements. If you have an external audit, you might require specific documentation like attendance lists. An annual report tells part of the story, but you need to consider if it is sufficient. You might need more frequent reporting or other tools like field visits to confirm what partners are doing.

00:51:16 Q&A: Theory of Change vs. MEAL plan

Question: What do I have to set first: Theory of Change or the MEAL plan?

Answer: The Theory of Change is the first step. It identifies the long-term objective. Without the long-term objective (macro level), you cannot proceed with the MEAL plan, which operates on a more micro level.

00:52:08 Q&A: When to develop the MEAL system

Question: When should the MEAL system be developed?

Answer: The development of the MEAL system starts at the project proposal stage. The first bits are located in the design of the project. When you start implementing activities, you recap what was in the proposal and ensure everything aligns and makes sense now that resources and people are in place.

00:53:07 Q&A: MEAL and quality assurance

Question: What is the relationship between MEAL and quality assurance?

Answer: Quality assurance is a process of guaranteeing high quality in your work. Having a good MEAL system in place is a way to guarantee that you have high quality assurance within your project.

00:53:47 Q&A: Communicating the MEAL plan

Question: How to communicate the MEAL plan to all project team members in an easy way?

Answer: The programming team and MEAL team should create the plan together to ensure common understanding. Then, you can include people who need to be aware via different facilitation methods, such as presentations, games, or training sessions. Ensure the presentation style applies to everyone, especially if including field staff with different levels of responsibility.

00:56:37 Q&A: Managing multiple country programs

Question: Can ActivityInfo be used for managing performance of country programs with multiple projects and implementation partners?

Answer: Yes, it is a frequent use case. It facilitates management when you have many projects, countries, and partners, and supports headquarters in getting information easily.

00:57:32 Q&A: Standardization of MEAL plans

Question: Is the MEAL plan standard for all, or can we develop one according to the needs of the project?

Answer: The MEAL plan always corresponds to the needs of the project, program, and specific countries. Even two identical projects in different countries will likely have different MEAL plans due to the context.

00:57:54 Q&A: Tools for qualitative analysis

Question: What are some common tools for qualitative analysis?

Answer: The most frequent case for qualitative data analysis is coding. This can be content counting, open coding (without predetermined themes), or coding with predetermined themes.

00:59:23 Q&A: MEAL plans in emergencies

Question: Is a MEAL plan enough to monitor an emergency project?

Answer: No, it is not enough. While you speed up the process, you will need other tools like the feedback flowchart to respond to beneficiaries, a learning plan, or a communication plan. These tools accompany the MEAL plan to make the system work efficiently.

01:03:01 Q&A: Missing data during evaluation

Question: During an evaluation, if it is found that data on relevant indicators has not been collected, how does an evaluator proceed?

Answer: You need to ask what kind of data is missing and if there are other sources. You might be able to go for fresh data collection to address the gap, perhaps using a representative sample to generalize. There are mitigation measures, but it depends on the type of data, compliance requirements, and availability of other sources.

01:04:22 Q&A: Reviewing the MEAL plan

Question: Who should participate in the review of the MEAL plan?

Answer: The programming team (who know the context and challenges), the MEAL team (technical expertise), and the Information Management teams. You may also want feedback from field staff to identify issues.

01:05:25 Q&A: Community participation

Question: How do we ensure community participation in MEAL?

Answer: Community participation starts with the assessment phase by including project participants in building the project. To the extent that indicators aim to get feedback from communities, we ensure we get beneficiary feedback to adjust activities and MEAL activities.

01:06:32 Q&A: MEAL plan vs. learning plan

Question: What is the difference and similarity between a MEAL plan and a learning plan?

Answer: The MEAL plan uses information horizontally (e.g., X% agree), while the learning plan uses information vertically (e.g., to what extent does agreement contribute to the next level of the Results Framework). The learning process may require additional data collection not included in the MEAL plan. If the learning plan ends up completely different from the MEAL plan, you need to check if the MEAL plan makes sense.

01:10:51 Q&A: Data protection

Question: What are the best practices to protect vulnerable communities' data during data collection?

Answer: Ensure you don't gather information you don't need. The MEAL plan supports this by filtering necessary data. Consider how you collect information, where it is stored, who accesses it, and how long it is kept. When sharing information, filter with whom and for what purpose to ensure you don't expose beneficiary data to people who do not need it. Secure storage and access control are crucial.