00:00:00 Introduction

All right! Thanks, Fay. Again, I am very excited to be able to share this today. With such a big group, we're really overwhelmed by the response, so welcome to everyone. We are looking today at planning and implementing data collection systems for monitoring and evaluation.

If you look at the whole process for M&E, this is a concern that begins once you've determined what you're going to measure in your program. Now, deciding what to measure is a much bigger and arguably more important question than data collection. You might be using the theory of change methodology or a logical framework to monitor and evaluate your program, or you might be relying on sector-specific approaches to measurement like health, nutrition, or education where indicators and measurement are well defined.

At the end of the process, of course, the whole point of M&E is to be able to ultimately provide decision support so that you can improve programs in order to deliver better outcomes for beneficiaries. But you can't get there without a solid data collection system, and that's what we're going to look at today. We're going to assume that you know what you need to measure, and we're going to focus on what you need to know about how to collect it.

Here is our roadmap for today. It's necessarily going to be very quick because we only have an hour, but at ActivityInfo, we've worked with many organizations to implement data collection systems. Over time, we've tried to distill this process down to a roadmap with seven steps, starting with an inventory of all of your indicators, followed by mapping out where that information comes from, the flow of information, designing the data collection forms, building the system, launching the system within an organization, monitoring it, and finally evaluating and iterating.

00:02:15 Step 1: Create an indicator inventory

We're going to start with the first step: the indicator inventory. In some ways, this is starting from the end and working our way backwards because we're going to create an inventory of all the indicators that we want our data collection system to produce. This is the end product that we're after.

To create this inventory, I recommend using a spreadsheet to simply list every single indicator that appears in your log frame, in your grant agreements, donor reporting requirements, and so forth. You might be responsible for multiple log frames for different projects and donors; it doesn't matter. Combine them all into a big list. You'll also need to speak with your management team. They may also have expectations that aren't listed in the log frame. For example, there might be a weekly management meeting where they expect to have updates on the progress of the number of sites that have started construction. That might not be in your log frame, but it needs to go in your list if you're going to collect it.

Once you have your list, you'll need to start thinking about the level of detail you will need, both in terms of how often you report on the data and how to break these indicators down. Let's start with breakdowns first. If one of your indicators is "number of schools constructed," for example, a single number will rarely be enough. Let's look at a few common breakdowns or disaggregations that are often needed.

The first is geographic. If your program is doing great in the north and terrible in the south, the average will look okay, hiding real problems. So, from a monitoring point of view, it's important to be able to show these differences, and this will affect your data collection system. Think also about mapping at this stage. If you want to have nice detailed geographic maps with all of your schools listed, for example, then you're going to need to have at least some of your indicators at the school level—a very detailed level.

The next two important categories are demographics and legal status. Think about having breakdowns between sex, age groups, disability status, or between IDPs, refugees, and host community members. These might be breakdowns for your own programmatic goals—for example, to support gender mainstreaming or a focus on those with disabilities—or they just might be hard reporting requirements from your donors. Either way, note these breakdowns in your indicator inventory table.

Finally, consider organizational breakdowns. These are often important for management purposes. If you work with multiple implementing partners, for example, you'll want to be able to monitor the progress of each independently. Within a single organization, you might need to track differences between field offices. I should note the breakdowns might be different for each indicator. For some indicators, you'll need very detailed breakdowns; for others, a total might suffice.

For each indicator, you'll also need to determine how often you'll need to report, publish, or share the values. Do you need to report to your donor every quarter or only once a year? Which indicators does the management team expect to see every week or every month? It's also going to depend on the type of indicator. Maybe your impact indicators might only be collected once or twice (baseline and final) during the course of the program. It also depends on the nature of your program. In an extreme crisis, you might be expected to report morbidity and mortality rates every night or even twice a day, as we saw sometimes during the phases of the Corona crisis. For development projects, reporting frequencies are typically quarterly or annually. Again, this is going to be different for each indicator, so make a careful note in your indicator inventory.

00:06:30 Step 2: Map the information flow

With our indicator inventory at hand, we're going to move on to the next step: mapping information flow within your organization. I think one common mistake I see in planning M&E systems starts with the phrase, "I only need a few numbers." This can often lead to ignoring the complexity behind the data collection process. It's important to really dig down into each indicator and understand where that number is coming from. A single number might require someone somewhere to collect thousands of data points.

If we take an example from our inventory, let's look at an indicator like the number of women who benefited from non-food items. If you start asking, "Where does this number come from?", the first thing that you might find is that no single person in your organization has this number. You might have to go to three different field offices to get the answer. And you can't stop there; you have to ask how did these field offices come up with the number for the indicator?

You might find that in one field office, they need to go back to actual paper-based notebooks that people sign during the distribution to come up with the gender breakdowns. In a second field office, you might find that people are taking laptops to the field and entering beneficiary information in a spreadsheet as they release kits to families, so that might be the ultimate data source. In the third field office, you might find that they're not collecting this information at all, and that's something that you will need to address in your data collection plan. Otherwise, you can't be sure that you're getting reliable data.

The other thing you should be asking is which team and who exactly is providing the data. In some cases, all of your results might come from a single team that is funded by the program's donor. In other cases, you might have separate teams carrying out different activities. Each field office might have one team doing distributions and another construction. This means that you'll need to collect data from two different sources. You should do the same thing for the qualitative indicators in your inventory. Often these are the results of expert judgments, so you need to identify the person and the team who will be making that judgment because she or he will be your data source. They might in turn rely on lengthy evaluations, interviews, or focus groups of their own, but from the point of M&E, you also need that judgment as to whether the indicator has been met.

00:09:15 Step 3: Design the data collection forms

Two down, five to go. Step three. All right, now that we've listed what we need and we've mapped where it's coming from, it's time to start designing our forms for collecting the data. Now, I say "forms" because that's how data collection is structured in our software, ActivityInfo, but this could just as easily be the structure of a spreadsheet or a paper form or something else depending on the tool you're using. We're talking now about the structure of your system.

The first choice, and this is key, is the choice as to which level your M&E system is going to be collecting data. You can see that there are already systems in place in the field to collect raw data. The distributions teams simply wouldn't be able to do their job without beneficiary lists, so some sort of system must exist, even if it's on paper. One option is to leave those systems alone and just collect the totals from the field offices every month or every week. We call this periodic reporting.

This is pretty common: you send out a spreadsheet each month to the field offices or use a web-based tool like ActivityInfo where the teams in the field fill out a monthly report online. Using an online form can save a lot of time and frustration compared to emailing spreadsheets back and forth, but it's still asking people to enter data twice: once in their own systems that they use in the field and a second time to send you the numbers for your system.

So, another option is to push the M&E system all the way down to the field. Rather than each office using its own homegrown system, each field office records the distributions in a centralized database—maybe with a tablet working offline, synchronizing right at the field site. This way, the indicators you need for M&E can be calculated automatically. You don't have to chase after people every month to send in their reports because you have the information as soon as they do. In most cases, I really do recommend pushing data collection as far down to the field as possible. However, if you're working with a large number of external partners, that might not be realistic. In that case, periodic reporting might be the best choice. Getting 20 or 100 different organizations, each with their own internal processes and methods, to use the same day-to-day system requires more effort and time than you may have for your M&E system.

Once you've decided the level of data that you're going to collect, it's time to start organizing your forms. When you're setting up your system, you want to make sure that you're structuring your data collection forms around who is providing the data and when. I want to emphasize that the way that you organize your data collection is usually going to be different than how you organize your analysis. Sometimes I see people trying to structure data collection in the same way that their logical framework or analysis is structured, but that's not necessarily the best approach. You can always present your data in a different way at the end, but your data collection should be organized around how data is collected.

If we're looking at who is doing the reporting, it makes sense to have a separate form for each team. That way, the teams only see the indicators that are relevant to their work, and they know exactly what they need to provide. You will also need separate forms for different reporting frequencies. If you're collecting different information on a weekly basis and other information on a quarterly basis, then it makes sense to add two different forms. Likewise, if you're collecting indicators at different geographic levels, it also makes sense to separate them into separate forms. For example, if you have a village assessment, your team is going to have to complete one form per village. If you have health zone performance statistics, that's going to require a separate form because it's a different unit of analysis.

Having said that, you don't want to go too far and end up with hundreds of forms. You don't want to have a case where you have one form per indicator. Think about how you can combine forms together to make it a little bit easier for the people reporting data. One of the things that you can do is combine forms that have similar indicators. For example, you might have three education activities: distributing textbooks, classroom rehabilitations, and teacher trainings. Maybe you have different teams doing those activities, but if they have mostly the same indicators—like number of students, location, and the same frequency—then it might make sense to combine them into a single form for education interventions. Then you can use relevance rules or skip logic to show only the indicators that are relevant for those interventions.

00:14:10 Step 4: Build the system

Now it's time to start building the system. Obviously, building an information system deserves a whole course in itself, but for today I have just enough time to highlight a few factors at the technical level.

The first and most important thing is the flexibility of the system. In the weeks and months after you launch your system, it will need to constantly change. Stakeholders will demand new information, your program may change and adapt as the context changes, and you're going to have to make changes to your data collection system. If your system can't quickly change, people will stop using it. That explains why Excel or spreadsheets dominate so many programs; it's very easy to change an Excel spreadsheet. A centralized database can be more efficient and powerful, but if it's not easy to change, it will get abandoned.

The second thing is data quality rules. Do not assume that the people participating in your system will be as precise as you would like them to be. I learned this the hard way. In the first system I built for UNICEF, we didn't have a complete list of villages. We let people add new villages if they couldn't find theirs. People did not check the list; they just added a new village. So you had Goma, Goma with an E, Goma with an A. We had thousands of villages after the first couple of months, and it was impossible to do any kind of analysis because of the duplications. If you're working with more than two or three people, you have to nail down everything. Make sure you have a predefined list to the extent that you can. Add validation rules so that people can't enter a million beneficiaries. Do as much as you can when you're building your system because chasing people down afterwards is much harder.

The next thing is user permissions. The minute that you have more than five people contributing data, you need to have a way of regulating what people can change. This is related to security, but more importantly, it's to keep the system from descending into chaos. People are very busy in the field and have a tendency to make mistakes, like overwriting somebody else's results. Make sure that you tune permissions precisely so that each person in your organization can only edit what they are meant to.

The last thing on my list is offline capability. Not everywhere we work has perfect internet connections. Synchronization is a hard technical problem. If you're in a situation where you're exchanging data with the field, make sure that you have a good solution for offline work. That can be choosing either a simple solution where the data flow is just one-way from the field to headquarters, or something like ActivityInfo where we spend a lot of time on synchronization to make sure that data can be synced both ways.

00:17:20 Step 5: Empower the data collection process

I would love to spend more time talking about building M&E systems, but there really are more important issues when it comes to building data collection systems, and that's the people factor. You can't focus exclusively on technology because people really make the system work or not work. Let's say that you've built your system, the technology is sorted, and you're ready to go. Now you just need people to use it.

I'll start with something boring but very important: human resources. The best technology in the world does not make data collection free; it is going to cost time. Make sure your program's budget includes staff for data collection and that job descriptions are right. Who in your organization is going to be responsible for sending and collecting this information? If you're working with external partners, write the data collection and reporting requirements into the grant agreement or the contract. This ensures they know they have to plan for it and have somebody on their team who can take the time to ensure good quality data.

The next thing is to spend time on training and documentation. Take the time to write a good step-by-step guide specific to your organization and program. Hopefully, you choose a user-friendly system, but people still need to know when they need to report and what specific indicators mean. All of those details need to go into a good guide. You can back that up with workshops, Zoom calls, recorded webinars, and YouTube videos.

The last thing, and I think this is the most powerful thing you can do to make your system succeed, is to share results early, often, and widely. When people see the information from the system being used and that it actually means something, they will see the value of the system and be far more motivated to participate. I remember working with the NFI cluster in Eastern Congo. When we had our monthly meetings, we would put a map of distributions up on the projector. It was super useful for coordination to see gaps, but it also provided a huge incentive to report because nobody wanted to see that map without themselves on it. If people don't see anything come out of the system, it's a hard sell to get them to participate.

00:20:05 Step 6: Monitor the data collection process

We've come to step six. You've built your system, it's well designed, and you've launched it. But you're not done yet. M&E systems require a lot of monitoring and follow-up.

The first thing you should be doing is reviewing the data as it comes in. Actually look at the data that people are putting into the system. This will give you an opportunity to spot problems early and add more data validation rules. If you see a lot of typos in refugee registration numbers, for example, add an input mask. If you see data that's missing, it gives you a chance to go back to the partners and find out why.

The second thing is to lock data that's been validated and published already. That way, the numbers from last month that you've already sent to your donor don't change unexpectedly. You want to have as few opportunities for mistakes as possible.

Finally, it's very helpful to establish a regular reporting rhythm, especially if you're doing periodic reporting. For example, say that every month on the 5th is the deadline for reporting from the previous month. On the 10th, five days later, that's the deadline for feedback and corrections. Five days after that, reporting closes. This helps everyone know what the schedule is.

00:21:40 Step 7: Evaluate and iterate

We're at the last step, which is really a continuation of the whole process. We started with the indicator inventory, but in reality, it can often be very difficult for people to describe what information they need. Often you'll need to start with some basics, and then as soon as people see it, they will realize what they actually need.

What you learn from your data collection systems is going to inform your log frames, your donor reporting, and your theories of change. You are going to learn a lot from the act of operationalizing the data collection system. It might be small things, like realizing there was no clear definition of how many latrines were being built—some partners measuring doors, others measuring blocks. You have to go back to your logical framework and make those precisions.

In other cases, you might have put indicators in your log frame that aren't possible to collect. Maybe you counted on a survey or regular data coming in from the field, but because of security or access problems, you can't actually get that number. Then you have to go back to the beginning and look at how you are going to effectively monitor or evaluate the program.

As soon as you start showing results to your management and stakeholders, you're going to get feedback from day one. Someone is going to say, "That's not what I need for my program." That's the moment where you have to use that feedback to go back and incorporate that into your log frame or your data collection system.

00:23:30 Q&A session

What is the best way to approach building institutional rather than project results frameworks? Is it top-down or bottom-up?

That's a great question. In my experience, you have to meet in the middle. You can't ignore either one. I advocate for using the indicator inventory step and throwing everything together in a list because that's when you're going to start to see where the overlap is between your high-level goals and your activity reporting. If you're looking at a global level, maybe you can't do this with all 70 programs, but take your global results framework and the log frames from a couple of activities or projects, combine them, and start to look where there's overlap. You'll see things from your high-level goals that are missing from your activities, which you'll have to introduce to the field. You'll also see that you might have to be more precise about some of your high-level goals.

How to establish a quarterly evaluation system for ongoing projects?

For better or worse, often we don't have a chance to put together M&E systems right from the beginning of the programs. I would use this approach for existing programs in the same way. Start by looking at what you need to measure for your quarterly evaluation system. Figure out where that information is available currently. Start with a data collection form—ideally an online form—that you can send everybody every quarter. Build it, share it with your colleagues, and go from there.

The COVID-19 pandemic saw a huge change in how to collect data. Any insight on how to best adapt M&E work to this kind of situation?

I think we're all dealing with this. One thing is training; we've moved from in-person trainings to virtual formats. It's also made local participation in systems more important than ever. As staff movements are disrupted, people who might normally be based in the field are now stuck elsewhere. It helps to empower data collection at the lowest possible level. If you have a situation where the people doing the work have ownership of the information system, you're going to be in a much better position. I really encourage you to push the data collection systems down as close to the field as possible. It improves data quality and makes them more resilient to disruptions.

Is designing the log frame the M&E specialist's role or the program specialist's role?

You need both people at the table. I have experience designing information systems, but I am not a nutritionist or an expert in agriculture. We need to rely on domain expertise. But in turn, those colleagues need to be guided by you with your expertise in systems and processes. A nutritionist might say the best way to measure impact is wasting in children under five. As a systems specialist, you have to ask: How do we get that number? Is there a health system in place? Is it feasible? It needs to be a dialogue.

What is the difference between a data collection tool and a data source?

It's important to map out where this data is coming from. When I say "data source," I'm talking about where your system gets that data. You might have multiple systems. For example, a registration logbook in the field is a system, and that becomes your data source. The tool might be the notebook they use. If you have an M&E system built on top of this, the logbook is the source, and maybe the spreadsheet you send out once a month is your data collection tool. We try to reduce the number of different systems so that you don't have to keep moving data from one to another.

How do you adjust indicators without negatively affecting data quality?

Change is constant. One strategy, for example, if you decide you need to track disability status halfway through a program, is to add the new field but use a calculated field to estimate results from previous months based on known statistics. It's not perfect, but don't let the perfect be the enemy of the good. Use the data you have available to make estimates moving forward.

Can you go into more detail about finding the balance between too many forms and keeping forms simple?

It depends on the system. With ActivityInfo, you can add logic to forms to show only relevant indicators based on the partner or activity. The trade-off is the amount of work you put into designing the system versus the user experience. If you have a large group of users, that extra time might be worth it. In some cases, like humanitarian response plans with many sectors, you will have a lot of forms. In that case, organizing forms into folders and using user permissions so people only see what they need can solve the problem.

Any insight on how to best merge donor and internal requirements to avoid indicator overload?

We all suffer from indicator overload. The indicator inventory helps you evaluate the costs of data collection. Data collection is expensive. If you pull all indicators together and have 200 items, you need to push back. Do an estimate of how much it will cost. If you go to your project manager and ask them to fill out a spreadsheet with 200 indicators, it's not going to work. By assigning costs, you can provide a rational answer to stakeholders: "If you need this information, I'm going to have to hire an assistant. If not, we have to make choices."

What is the basic difference between monitoring and evaluation?

Monitoring is about steering a project or program and making adjustments on a frequent basis. Evaluation helps you learn for the next project. If you wait until the end of a three-year program to find out nobody used the textbooks, it's too late. Monitoring provides steering; evaluation provides learning.

Do you have advice for qualitative indicators in an online system?

Structuring open-ended data can be very powerful. Instead of just having a text box that says "Anything else?", try adding specific open-ended questions about themes. For example, "Please write a paragraph about the participation of beneficiaries in the system design" or "Please provide a paragraph on the impact of security on the program." This provides a bit more structure than a generic text box.

How can we make the M&E system effective in measuring qualitative change, like attitudes and behaviors?

It can be useful to use a two-stage approach. Spend time doing qualitative, open-ended research (interviews, discussions) to identify specific attitudes and behaviors. This requires people with social science backgrounds. Once you identify the attitudes, you can turn that into a quantitative tool that can be applied more widely and measured over time. You can then use statistical methods like principal components analysis.

Thank you so much for joining. We will send the recording and slides out by email.