00:00:04 Introduction

Thanks so much for that introduction, Faye. Hello, everyone. Thanks so much for joining us for today's webinar. I'm very excited to be talking to you about today's topic because it's something that I've lived through in my past experience as an M&E officer. I'm happy to be sharing some insights I've learned from those experiences with you today and also from our learnings working with NGOs of all shapes and sizes in the years that we've been supporting our customers here at ActivityInfo who are dealing with these questions about how to make M&E systems work. Hopefully, today we can share with you some insights on what we've learned and you can take something away that you can apply in your own organizations or for your own projects.

The agenda is quite simple. We'll start off with answering a very basic question of why we should even think about assessing our M&E system to begin with. Secondly, we'll go into the practical matters of how you can actually conduct an M&E system assessment for your own projects or programs. Then we'll go into what to do following an assessment. And then, of course, as always, we'll leave some time at the end for Q&A.

00:01:32 Why assess an M&E system

To start, I think it would be useful for us to recap what we actually mean by an M&E system. Our understanding of an M&E system refers to the entire set of the people, the processes, and tools that work together to enable the collection, analysis, and use of monitoring and evaluation information. To put it simply, it's the collection of all of these activities and components that help us to achieve what we need to do within monitoring and evaluation at the end of the day. We've done a number of webinars in the past that go into the details of each of these components of an M&E system. If you check out our website, you can see a full webinar series on that topic if you want a deeper perspective on those elements.

With this foundation, let's answer the question: Why assess an M&E system? To put it simply, we want to know if it's working. We want to ensure that our M&E system, that we spent so much effort and time building, actually works. We want to make sure that it works so that it can fulfill its dual role of learning about our projects and our programs, and providing accountability to the people we're hoping to serve as well as to those who provided the resources to enable the project to begin with.

Practically speaking, this means three things. First, around implementation, we want to know whether the system components that we said we needed, or at least we think we should have, actually exist and are being implemented as we expect them to. Secondly, this means that the implementation of these components is actually achieving the results we hope to achieve in terms of collecting, analyzing, and using M&E information. Thirdly, if we find that those results aren't being achieved, an assessment helps us to know in what ways we can actually strengthen the system in order to achieve those results in the future.

00:04:46 How to conduct an M&E system assessment

There are a number of steps that you'll need to do. The first step is to determine the scale. It's important to understand that M&E systems can exist at different levels. At the most basic level, you can have an M&E system defined for a specific project measuring your project results and indicators. At the next level up, you might have a program that is a combination of multiple projects contributing to some program goal. You could also have a system implemented at the organizational level, measuring against an impact framework to see the results that your organization as a whole is achieving. Finally, at the top of the scale, you might have a system that exists at the national level, involving comprehensive systems that collect data across different actors.

It is important for you to decide in the first instance which scale you are conducting your assessment in. Depending on the scale, there are different questions to ask and different considerations. For example, at the organizational level, you'll need to consider the organizational structure and strategy. At the national level, it will be important to consider environmental context factors such as the political situation, culture, policies, or legislation. Doing an assessment at the national scale will be the most time-consuming as it requires coordination across multiple entities.

Once you've identified the scale, the next step is to determine the scope. Scope refers to the extent of the system that you wish to include in the assessment. To understand what is included, we can look at a helpful framework produced by FHI 360, where they've identified twelve domains of a functional M&E system. On the people side, we look at HR capacity, management, and M&E leadership. On the process side, we look at specific procedures and systems for data collection and organization. With regards to tools, we look at technology for analysis, quality assurance, and organization. Other domains include budget, conduct of evaluations, and how learning is disseminated. To have the most comprehensive assessment, it would be useful to look at all twelve domains, but if you have limited resources, you can choose a subset.

The next step is building the team who will be conducting the assessment. In most cases, the core assessment team will be comprised of the managers who are ultimately accountable for the delivery of the M&E system, such as an M&E manager. In some organizations, you might have a data manager or an information management manager. If you have all three, that would be your core team. In addition, project staff and implementing staff who are involved with the actual delivery of your project or data collection in the field are important members as they provide inputs and qualitative feedback. In some cases where independence is required, you may engage a third-party consultant or firm to lead the assessment.

Following building your team, you have to determine the timing of your assessment. It would be useful to do an assessment at the start of the project to establish a baseline. This allows you to understand the current situation and identify missing critical elements. From there, it is useful to do a regular reassessment, typically on an annual basis for multi-year projects. If you identify significant gaps, you might choose to do it more frequently, such as every six months. If you join a project later and missed the baseline, do an assessment at the earliest opportunity. Finally, link the timing of your assessment to the budgeting cycle so you can prepare a business case for any necessary investments.

Now we get to the actual matter of conducting your assessment. This is usually done in a participatory process involving stakeholders. You will usually want to do a combination of three data collection methods: an extensive review of documentation (theory of change, MEAL framework, results plan, indicators, reporting tools), a series of interviews to capture qualitative insights, and potentially a survey for larger teams to gather quantitative data. The combination of these insights will lead you to an overall performance rating. This rating can take the form of a quantitative score accompanied by a qualitative explanation.

A useful tool for this is the Monitoring and Evaluation Systems Assessment Guide produced by FHI 360. It provides a matrix with specific questions for each dimension, usually worded as yes/no, though you can provide partial scores. The group of stakeholders should come to a consensus on the rating and provide a rationale.

The final step is putting together your assessment report. This report should contain a description of the methodology (scale, scope, methods), a discussion on your findings (performance rating, context, strengths, and gaps), and an action plan. The action plan should contain a list of recommendations and specific interventions, identifying who is responsible, how they will be done, and the relevant timelines.

00:29:38 What to do following an assessment

Let's explore some potential actions you might identify in your action plan. On the people side, you might find missing key roles, such as someone for data quality assurance, requiring recruitment. Or, you might find knowledge gaps requiring targeted training. On the process side, there might be ambiguity in tasks, requiring the creation of documentation or standard operating procedures (SOPs). In terms of tools, you might need to update systems, develop new tools, or create information products for dissemination. Finally, there might be a gap in your technology stack requiring investment in new hardware or software.

It is important to build a strong case for investment to secure the resources needed for these interventions. The way to do this is to conduct a Return on Investment (ROI) analysis. Costs might include staff wages, training costs, external experts, hardware, or software subscription fees. On the flip side, returns can be measured in time saved in collecting and analyzing data, increased productivity, increased learning, and increased satisfaction among staff, managers, and donors. If the potential return is greater than the estimated cost, you are in a good position to build a case for investment.

00:38:26 Case study

I wanted to conclude this webinar with a quick case study drawing on a personal example. Before joining ActivityInfo, I was an M&E officer at the Pilipinas Shell Foundation, a local NGO in the Philippines. I was brought on with an organization-wide mandate, and at that time, there wasn't a formal organization-level M&E system. The first thing I did was a baseline assessment at the organizational level. I measured indicators illustrating the strength of key M&E capabilities through document review, interviews with program managers and M&E officers, and a staff-wide survey.

Following that baseline assessment, I identified several gaps. First, there were unclear responsibilities regarding who was responsible for submitting and consolidating data for annual reports. Second, there were inefficient reporting processes with data scattered in different tools. Third, there were missing M&E elements, such as a well-defined theory of change for many programs. Finally, there was poor data quality due to the lack of a centralized information system; data was scattered across different Excel files, often leading to data loss when staff left.

With these gaps identified, I put together a concrete action plan. To address unclear responsibilities, I drafted formalized roles and responsibilities to be incorporated into job descriptions. I put in place updated reporting procedures for tracking key organizational indicators. I launched a project to roll out theories of change for each project, along with capacity building. Finally, I launched a procurement process for a software solution to serve as a central information management system, which ultimately led the foundation to use ActivityInfo.

00:45:11 Q&A session

What is the best approach or tips to identify people stakeholders during the assessment based on the scope we choose?

It's important to get clarity on the scale and the scope of your assessment first. If you're doing it at the national level, you need representatives from all implementing organizations. The general rule of thumb is to have representation from the groups who will be most directly impacted by the success of your M&E system. You need representation from your program team, inputs from people collecting data on the ground, and insights from people building the information products. Ask yourself who would be most impacted and ensure you get at least one representative from each of those groups.

Will you conduct another webinar on how to use the findings of an assessment?

That's a great idea. It might be useful to have a webinar on the practicalities of implementing the interventions identified and building M&E systems. We will consider that in our planning for upcoming webinars.

With the methodology and the score, can we have all these questions to assess the different scope?

Certainly. One of the resources I mentioned, the FHI 360 assessment tool, contains a long list of specific questions across all twelve dimensions. It was built with health programs in mind but is flexible enough to adapt to different kinds of programs.

Can you elaborate on various data collection methods to confirm authenticity?

I would recommend implementing the full range of data collection methods if you have the resources. It's important to have both quantitative and qualitative data points to triangulate results. A document review shows objectively if something exists, like a MEAL plan. However, an interview with a field worker might uncover practical limitations, such as a lack of validation mechanisms in a form leading to erroneous values. These nuances come out if you invest time in the full range of methods.

What is the optimal timeframe of an assessment for a large organization, like a UN agency in a specific country?

It depends on the size. At Pilipinas Shell Foundation, with about 50 staff, the baseline assessment took six to eight weeks. In a larger organization with many more stakeholders, it could take a number of months. You want to balance rigor with timeliness. If there are grave gaps you observe immediately, you might want to do an expedited assessment focusing on key domains in a few weeks so you can launch straight into interventions.

I am a new M&E practitioner joining a local CBO. A results framework is in place. Do I need to design a new M&E system or results framework?

I would say do the assessment first to determine whether you need to do a redesign. I wouldn't launch into redeveloping your system right away just because you're new. Your assessment might find that there are a lot of strengths in the existing system. It's always easier to build on something that is already working well than to build something entirely new. Take stock of the strengths and weaknesses first.

Should the baseline and end-of-project assessment be done together with the project baseline and endline evaluations?

You want to make sure you have the resources available to support both initiatives. Generally, you want to position the system assessment as close as possible to the start of the project baseline. However, if resources are limited, prioritize the actual project baseline data collection first. Once that is done, you can think about the assessment of the M&E system thereafter.

How is ActivityInfo connected to this discussion?

ActivityInfo is highly relevant. One of the key domains of an M&E system is the technology and systems in place for organizing data. An intervention might be to invest in technology to manage data more efficiently. ActivityInfo is an end-to-end solution for managing M&E data, spanning collection, analysis, and use. If you find a gap in your technology stack, ActivityInfo could be the solution. We also have resources to help you calculate the ROI of investing in ActivityInfo.

How often should we do this kind of assessment?

For a project spanning multiple years, the recommendation would be to do it on an annual basis. In long-term projects, much changes in a year—staff depart, contexts change, or funder requirements shift. You need to reassess to ensure the system is still fit for purpose. If there are grave gaps that need to be addressed quickly, you might want to do it more frequently to ensure interventions were successful.