00:00:04 Introduction and agenda

Thank you Fay for the great introduction as always. Today, we will take a look at the global M&E system. We will see what the purpose is and how complex an M&E system can be at the end of the day, as well as the characteristics of a complex system. We will move towards how an information management system can support and implement the needs of a global M&E system and the importance of the environment.

Then we will go into organizational maturity. Here we will see different case studies based on our customers and how their environment—meaning the structure and different processes and policies in place—affected the adoption of a global information management system. Finally, we will look at how ActivityInfo enables the global M&E system implementation, followed by a few minutes for Q&A.

00:01:16 The global M&E system: purpose and complexity

The reality is that there are many methods and processes that work well within straightforward, time-bound projects, but do not work well necessarily in large programs or across entire organizations, or at least work in a different way. This false assumption has created a few challenges. First is the unrealistic expectations regarding what an M&E system operating at a program or organizational level can actually achieve over a given period of time. Secondly, due to the lack of support or guidance by the organization in specific areas, performance has often been poor and many systems have failed to live up to their full potential.

Let's start with a quick reminder of what we mean by an M&E system. There is no single definition of an M&E system, which is why sometimes you will hear the word "framework" used as an equivalent. For the purpose of this webinar, I will define it as the people, the processes, and the technology that are involved. At the same time, this is the environment in which an M&E system operates. We include people at all levels: people from whom we collect feedback and people that help set up the system. We have many policies, M&E tools, and systems in place to implement them, and of course, we have the technology we use.

At the end of the day, the M&E system follows the data management circle. We collect a lot of information, we analyze this information, and the ultimate objective is to use this information timely for evidence-based decision-making.

There are many factors that influence an M&E system. The context is crucial; a system should be designed to serve a specific purpose, which almost always includes supporting basic project management, but may also include learning, accountability, and stakeholder needs. The level is defined as a place where plans are made and information is collected, analyzed, summarized, shared, or used. For example, an international NGO might work in several regions, each containing different countries, and at each level, plans are made and information is used.

Stakeholders and organizational structure also vary. Stakeholders may be external or internal, local, global, or regional. How teams are structured, where decisions are taken, and how external factors affect implementation are crucial elements that affect how the M&E system is designed. Policies and processes are crucial to the personalization of an M&E system to standardize how information is gathered, stored, analyzed, and disposed of. Finally, the type of project or program matters—whether it is a one-year project, a set of projects, or implemented in various countries.

00:06:05 Simple vs. complex M&E systems

Broadly speaking, we move from a simple case to a more complex case. On the simple end, we have time-bound projects implemented by a single organization. Moving towards complexity, we find projects in difficult or uncertain settings, programs run by large consortiums, or organizations working in a single location or sector. We then move to multi-phase programs implemented through multiple partners, and finally to the ultimate level of complexity: very large, multi-country, multi-sector international NGOs.

The global M&E system has serious differences compared to a simple M&E system. First is the scope and purpose. Complex M&E systems often need to interact with many other functions and processes, including knowledge management, financial management, and IT. Coordination across different departments becomes essential. Regarding purpose, while a project M&E system defines a primary purpose early on, a complex system faces competing demands from different levels—some interested in accountability, others in resource allocation, or advocacy. A complex system needs to be flexible enough to cope with constant change and shifting priorities.

Regarding levels, a project M&E system usually considers one or two levels. Complex systems operate on numerous levels: regions, countries, programs, and projects often implemented through partners. Information collected at one level often needs to be analyzed, summarized, or used on a higher level. Decentralization is also a factor; within a complex system, projects and countries need to operate independently but engage with each other to create a unified system. The job of a complex M&E system is partially to provide a framework within which different levels can function, balancing common policies with local flexibility.

Measurement of change is another major difference. In complex systems, it is much harder to define and measure change at an organizational level because organizations have multiple, sometimes conflicting, goals and do not operate under a single timeframe.

We need to balance linearity with flexibility. A simple system often follows an "if-then" relationship, reflecting a theory of change or logical framework. In a complex system with higher uncertainty, we need to keep linear models flexible to incorporate new needs or context changes. Technology and policies are mechanisms that help us strike this balance.

00:13:13 Key priorities for a global M&E system

Given these factors, I would highlight some key priorities. First, we need increased coordination and clear roles and responsibilities as we operate in a broader environment. We need a clear knowledge management framework, meaning policies and procedures for data collection, sharing, storage, and analysis. We need a supportive organizational culture where the organization defines the purpose of the system and supports its implementation through training, resources, and correct structure. Finally, we need a balance between the local level and the central level. Across all these needs, information is key to implementing the system.

00:14:43 The global information management system

Let's see how the global information management system satisfies these needs. An information management system helps manage data through the whole circle: starting from data collection and organization, moving to processing (editing, cleaning), and finally to analysis and use. All these processes happen within a context involving people, policies, and technology.

Regarding people, we have internal and external stakeholders at local, global, or regional levels. Policies provide guiding instructions on how each task is performed. We need to provide guidelines but understand they may need tailoring at the local level. In terms of technology, specifically information management technology, we must be conscious of decentralization. We need to adopt a system that can be institutionalized and used across different levels. Introducing a new system is a challenging task, so institutionalization is crucial. We must also consider integration with other departments like safeguarding or finance.

00:18:30 Leveraging technology with ActivityInfo

The key is to recall that equating technology to needs minimizes the resources required—people and time—allowing those resources to be used for key priorities on the ground. ActivityInfo provides a combination of flexibility and structure. It is a no-code relational database, meaning people can be trained to manage changes without needing specialized IT staff. Each database can be adapted to serve different needs at different levels, yet all live within a platform where they can interact.

This supports the data through the whole management circle, from collection to analysis. This brings efficiencies of scale; if we have many disparate systems, we need more capacity to manage them and more time for integration. A unified system reduces this burden.

00:21:08 Organizational maturity models

To understand the case studies we will discuss, it is important to analyze the Capability Maturity Model. This model, originally from the 1980s, relates to the degree of formality and optimization of processes.

We start at the Initial scale, where we pilot the information management system. The objective is not necessarily to get immediate value but to prove the value for scale-up. We move to Repeatable Value, where products are useful and implemented across multiple projects or countries. Next is Scaled Value, where the system is integrated into existing policies, and we identify people to spread knowledge. Then we move to Capable, where the role is significant in organizational decision-making, policy-compliant, and led by decision-makers. Finally, Efficient Scale, where efficiency is crucial, operations are actively managed and optimized, and we maintain what we have built.

00:24:43 Case studies: Overview

We have four different organizations: ACDI/VOCA, IREX, Lutheran World Federation (LWF), and Heartland Alliance. These organizations work in multiple countries, ranging from 6 to 100, with annual budgets varying from 20 million to almost 190 million USD. They focus on different areas: ACDI/VOCA on resilience and development; IREX on education and youth; LWF on livelihoods, protection, and services; and Heartland Alliance on mental health, gender equity, and justice. Their impact ranges from 50,000 to 3.5 million people served.

00:26:39 Case studies: Starting points and challenges

ACDI/VOCA had different databases and technologies that did not satisfy existing needs. They had to adjust quickly on the field but had different data sources, spending time on integration. IREX was using a system dependent on an external vendor, which created issues when they launched a new strategy requiring a new approach to strategic evaluation and learning. They needed to facilitate cross-program learning, and the external dependency made changes difficult.

Lutheran World Federation was also dependent on an external vendor where development stopped, forcing them to adopt a system that could be adjusted in-house. Heartland Alliance had many different systems used by each country, limiting the ability to access visualization and analysis for strategic decision-making.

00:28:40 Maturity and adoption speed

ACDI/VOCA and LWF started on the upper levels of the maturity model (Capable and Efficient). They had a clear vision, specific people leading the process, and policies in place, allowing them to adopt the system quite rapidly.

IREX and Heartland Alliance started by piloting and using the system on an ad-hoc basis. They gradually moved into building policies and putting dedicated people in place. This helped them move through the process to the "Capable" stage, where the system was integrated into policies. Both ACDI/VOCA and LWF had very strong executive sponsors—people with decision-making power to lead the process—which facilitated faster adoption compared to the more gradual approach of IREX and Heartland.

00:30:37 The role of the environment: People and teams

We operate within an environment of people, processes, and technology. Regarding people, ideally, we want strong teams at both HQ and country levels. At the HQ level, we need an implementation team familiar with M&E and data, along with executive sponsors. At the country level, we need strong implementation teams and local executive sponsors like country representatives.

For ACDI/VOCA and LWF, they had executive sponsors such as the VP of Quality or Global Coordinators, and dedicated project managers. IREX and Heartland did not start with a formal executive sponsor or dedicated project manager, which is why adoption took longer.

The composition of implementation teams varied. ACDI/VOCA included data scientists and directors; LWF included system officers and MEAL specialists. At the country level, teams included M&E officers, program managers, and country representatives. The key message is that we need implementation teams and people with decision-making authority at both HQ and country levels, with a good mix of technical and program staff.

00:34:29 Policies, processes, and standardization

A crucial factor for ACDI/VOCA and LWF's rapid adoption was having global project policy resources in place, such as detailed M&E plans, data management manuals, and policies on data quality. For IREX and Heartland, the adoption of ActivityInfo led to the development of global policies. IREX designed a comprehensive data management framework, and Heartland established procedures at the project level, incorporating M&E plans into proposals and budgets.

Standardized project-level implementation was also key. Organizations established a process for incorporating a program or country into the system. This typically starts from the proposal and approval, moving to the data model (the footprint for the database), database design, testing/piloting, migration (if applicable), and capacity building. This process usually takes one to three months.

For example, ACDI/VOCA ensures database design is driven by project needs and donor indicators, using stakeholder brainstorming sessions. IREX conducts rotation calls to introduce programs to ActivityInfo, followed by iterative design and testing. Heartland involves country MEAL teams working with the HQ MEAL advisor once a project is approved.

00:38:26 Capacity building and global rollout

Capacity building involves three stages: prior to adoption, launch, and adoption. Prior to adoption, we capacitate the design team, usually with vendor support, and involve executive sponsors. During the launch, we target Training of Trainers (TOTs) at the country level and build a resource library. In the adoption phase, we maintain refreshes, open office hours, and webinars. Training should be tailored, interactive, and hands-on.

The global rollout process involves assessment, pilot, and scale-up. Assessment involves understanding the need for change, aligning with strategy, and defining roles. Piloting involves choosing specific programs to test the solution and documenting lessons. Scale-up involves creating an enabling environment and establishing timelines.

Regarding speed, ACDI/VOCA and LWF adopted the system within approximately eight months. ACDI/VOCA included a pilot, while LWF assessed and scaled up directly. IREX and Heartland took longer, with IREX including a pilot phase. The crucial role here is the involvement of the executive sponsor and how structured the approach is.

00:42:40 Leveraging technology and data structures

An information management system has various levels depending on the data gathered: indicator level, activity level, secondary data, or beneficiary tracking. Organizations adopting a global system often gather the majority of data through primary data and beneficiary tracking, combined with secondary data.

We need to define the scope, roles, and decentralization level. As we move towards adoption, we often find ourselves gathering more primary data to be used by field teams. In terms of governance, we want centralized policies and guidelines but enable local actors to implement systems according to their needs.

In ActivityInfo, this structure often looks like different databases for specific countries, giving them flexibility. Standard information, like global indicator lists or master data, lives in separate databases that interact with the country databases. Some organizations use country templates as a starting point. These databases feed into global projects and risk/compliance dashboards to monitor delivery and compliance. We must clearly define entities, forms, and permissions, and use integrity rules like validation and relevance logic to guarantee data quality.

00:47:16 Conclusion

I would like to leave you with three key messages. The global information management system needs to serve the needs of the global M&E system: increased coordination, clear roles, knowledge management, supportive culture, and a balance between local and central levels. The system operates within a context of people, policies, and existing systems. Finally, we need technology to support flexibility, create efficiencies of scale, and support localization.

00:48:23 Q&A

How have these systems eased the day-to-day business role of project teams and field teams?

Imagine you work in-country and run multiple projects. If different projects use different systems, you have to go to different people to get information. If you have a common system, it saves time because you know where the information sits, you have access, and you can easily extract it in a similar format. Regarding strategizing and monitoring, a common system reduces the time needed for capacity building. If you have one system, you capacitate staff once, and that knowledge stays, rather than retraining for every new system.

To what extent were the projects humanitarian or mixed?

In the case of the four NGOs, ACDI/VOCA focused more on development, but in the sense of long-term projects. All of them work in the humanitarian sector depending on the needs. Recently, organizations work on a mix of projects. Many start with a humanitarian response and move toward development. Even in emergency response, having a common data point of truth facilitates speed. To deploy teams quickly, you need the institutionalization of a common tool so that staff on a roster can be deployed immediately without learning new tools.

What are the main pillars of a global M&E system?

If referring to the process of setting up the system, it should be a standard process similar to a project-based system but more complex. With more complexity comes the need for more flexibility. For example, project-based indicators might be strictly defined by donors, whereas a global indicator list needs flexibility—perhaps asking for the number of direct beneficiaries at a global level without requiring strict disaggregation that might be difficult for every program to provide.

What are the best practices in data collection in the context of crisis?

The most crucial practice is data protection policy implemented into practice. This means having specific procedures on how we treat, store, and collect beneficiary data, starting with consent. Data must be stored in a system with clear permissions so only relevant teams have access.

What are the elements to define clear data standards for a new project?

There are multiple standards. First is data protection. Then, data quality—organizations have checklists for avoiding duplication and cleaning data. Data quality goes hand-in-hand with data collection guidelines: how we reduce bias, how we phrase questions for protection, and how we conduct remote surveys. Standards differ depending on the scope and whether the data is quantitative or qualitative. These standards must go hand-in-hand with specific capacity building.