00:00:04 Introduction

Thank you. Faye, can you hear me well? Yes? Okay, great. So today we will see how the M&E system, the monitoring and evaluation system, is different in emergency response. What are the standards for the emergency response when we refer to monitoring and evaluation? And what is the difference between informal and formal monitoring as we need more readily available information in emergency response.

Then we will pass into how actually the information management system differs in emergency response. What are the steps of setting it up, what are some common challenges, and how we can overcome those challenges. Finally, we will use a case study, a practical example with the use of ActivityInfo when it comes to the information management system design, but also outlining a few resources that we use internally, not yet publicly available, in order to make the case and make the webinar a bit more practical. Finally, we'll have time for Q&As.

00:01:20 Defining emergency response

Starting by the definition of an emergency response. How do we define the emergency response? What does it mean? We have the cases that we have humanitarian crisis or disaster. UNHCR defines it as cases that threaten new forced displacement, cause loss of life or other serious harm, or situations that significantly affect the rights or well-being of displaced and stateless persons. Of course, in a country, we may have multiple emergency declarations at the same time if the crises are geographically or contextually different from each other. Example given is the case of DRC, the Democratic Republic of Congo, in 2021.

Also, emergency response can be defined by the fact that humanitarian crisis must demand exceptional measures because current governmental or UNHCR capacities are inadequate in order to provide an effective response and an effective solution. The emergency response, according always to UNHCR, has three different levels.

Level number one: we have the case that the operation country needs to improve the preparedness or start an initial response, but the current capacity of the country operation is insufficient. So you have the support from the region or from the HQ. In Level 2, we have a rapidly evolving humanitarian situation and here we face significant gaps in resources, staffing, expertise, and additional support and resources is needed. So in the first case still we need response, but we acknowledge that we can implement within the country. In the second level, we seek more response, more support from the HQ or the regional bureaus.

In level three, there is an exceptional serious situation. Actually, serious situation in terms of scale; it goes, it evolves pretty rapidly, it's pretty complex, and the consequences of the crisis significantly exceed the existing response capacity of the regional and the country bureau. So we have three different levels that differ, and of course, the commonality is actually that always an emergency response is something that is new, it's something that is evolving in a specific environment, and it's something that we should adjust the M&E system and the information management system to be able to fit data in order to enable us to monitor and look for the change in the specific context that we're working on.

00:04:14 Standards for monitoring and evaluation

What are the standards for monitoring and evaluation? I will start from the purpose of monitoring and evaluation in such a situation. The purpose actually is primarily can be grouped into two buckets. The first one: we need to assess the satisfaction with the response in all the processes, in all the evaluative processes. Second, we need to always have a formal feedback mechanism in order to capture both positive and negative feedback and suggestions from community members.

Evaluative processes now are generally useful in identifying recommendations for improving the next phase of the response or future responses. Whereas feedback mechanisms allow project staff to address immediately any issues raised by the community during the ongoing response, such as cases of inappropriate targeting and selection or staff behavior. Feedback mechanisms often include hotline numbers, help desks, community forums, and complaint boxes. A mixture now of these methods is usually appropriate given that community members may have different preferences about how to give feedback. It is important that those who do not receive support or participate in the response have access to those methods because these community members are an important source of information about the transparency and effectiveness of a project's targeting criteria and selection process, which is crucial at the first stage of implementation.

Monitoring and Evaluation plays a key role in maintaining two-way communication and feedback between project staff and community members. Both those who participate, as I mentioned in the project, and those who do not need to have access. It adds up as an addition to monitoring satisfaction with the quality of services or goods provided during the early response.

When it comes to the evaluative process, what we find more frequently is the real-time evaluations under an emergency response. So as the name indicates, an evaluation with the primary objective is to provide feedback in a participatory way in real time. So here we collect more in-depth information on the appropriateness, on the relevance, on the effectiveness, sustainability, the coverage, and coordination of the response. Usually, the real-time evaluation is being implemented six to eight weeks after an emergency response begins. The primary objective is also participatory and this is associated with the means that we choose to collect information. Usually, it's qualitative means. So most frequently the focus group discussions and the key informant interviews are being applied.

00:10:07 How is monitoring evolving?

How is monitoring evolving? We can separate into two different buckets: the phases of an emergency response. The immediate response, which is implemented at the first stage of the situation, is frequently characterized by limited staff, too little time, too much to do, priority focus on response, and fluid context. And then when the situation is being stabilized, often we have more staff, we have less pressure for immediate response.

So what is important is during that first phase we have a changing emergency context. We start delivering any goods and services throughout the implementation and the targeted population starts using those inputs, those goods, and those services. So there we start monitoring the output level which is frequently number of goods or number of services or what type of services we have delivered. And then gradually we pass into intermediate level monitoring. So a bit more in-depth monitoring on the satisfaction. So you receive the specific good or the specific service but are you satisfied? Did you receive it timely? So here we may find cases that we compare results between subgroups and we probe with "why" questions. Frequently an observation is crucial.

Then, as the situation is being stabilized, then we move to the strategic objective level. So there we may start considering how we can monitor and evaluate the strategic objective level. But also we may be in a situation that we need to add another more strategic objective. Example given, if you are working on emergency response and we distribute non-food items and gradually as we move throughout the months, after a couple of six months let's assume, we realize that there's another need. There's the need actually to satisfy the need for housing or implement referrals when it comes to housing because you have established collaboration with actors that are specialized in housing. There we may add another strategic objective that actually covers the next activity that we're going to implement.

Simple and flexible is tailored around the four Cs: count progress towards outputs, check the appropriateness and effectiveness, change response as needed, and communicate progress and results. So these four Cs when implemented efficiently provide timely information that is immediately relevant for maintaining a high-quality emergency response.

00:16:39 Formal and informal monitoring

Now what is different between formal and informal monitoring and why it's important? So for me, it's important the informal monitoring because when we find ourselves in a situation that things are changing rapidly, it could be the case that we don't have set up complete systems in place yet. We need still to be able to collect feedback and the collection of feedback can happen through the formal monitoring system, but also through the informal monitoring system.

Let me indicate what are the differences between those two types. First of all, the informal monitoring system aims to identify the changes. Whereas the formal monitoring system measures activity outputs, the immediate result, they count progress, they check the relevance and effectiveness. The informal monitoring can happen during all our field visits practically and by all field staff. The only thing that I need here is my staff to be aware that they need to be attentive on whether something has changed or whether there is an incident that needs to be communicated back to the office.

The formal monitoring is composed by different tools and has specific frequency and of course, it's being implemented by designated M&E staff. The informal monitoring happens as part of our normal interactions, whereas the formal monitoring is in a more structured way. Given the data collection tools that we use in the informal monitoring, we look more to find what we don't know, those unanticipated changes. In the formal monitoring, we are more conscious. We focus on the level of participation, on the timing, on the satisfaction needs, and access to services. And that is why if we compare to informal monitoring we have a very small purposeful sample. Informal monitoring can be feedback received by everyone.

Then formal monitoring, it's being used specifically on those daily debriefs and weekly debriefs. The formal monitoring can fit into the weekly debrief but also more in the more official debriefs, the monthly debriefs, the monthly coordination meetings. In both cases, we need to act immediately. That's the important thing in emergency response.

00:22:38 The information management system in an emergency response

Now the information management system in an emergency response. I will recap and I think that this is a graph that I have used in previous webinars and presentations but I find it very useful. So the information management system aims to control the information from the time that information starts existing, a data point, practical information. So this data point exists at the moment that we perform data collection. We organize those data points, we process—and by processing we mean the technical processes to transform data into a format that can be easily manipulated combined with other data in preparation for further analysis. At the end of the day, we analyze and use that information. So we produce real-time infographics, real-time reports, things that can be easily processed and easily shared with a broader audience.

The information management system allows for better decision making, it helps us better coordinate, and the timely data collection leads to faster decision making. And of course, it helps us think in the long term. So think that the project, the majority of the projects, they have a short-term window which is those first months of emergency response. But gradually they move more long term. So to this end, we need an information management system that can be flexible enough and allow for this long-term change.

Now the information management system lives within a context, within an environment. This environment is always composed by people: who is doing what within my organization? What capacity do my people have within the organization? Processes: data protection processes, data quality assurance processes, the monitoring and evaluation processes. Technology: existing systems, other systems in place, systems that my partners may use, other systems that I need to integrate with or I need to transfer information from.

The important thing in emergency is actually to be simple. So the more complex the solution is, the more places it can fail. One way to reduce the complexity of our information management system is to collect less data in the first place. We need to consider "need to know" versus "nice to know" information by ensuring that all data collected is linked to a specific purpose. This can be an indicator disaggregation, a point that leads to an expected programmatic decision, or another clearly defined use.

Flexibly quick solutions now may get the system up and running quickly. But as the response grows and evolves, the solution may not be flexible enough to incorporate new elements. As I mentioned, we may have a registration form, but it could be the case that at some point we want to add or associate another type of information. We find ourselves now in situations that we deliver goods, so it's fine if we have our registration M&E form that illustrates that service delivery. But it could be the case, as in the previous example, that we want to add this referral for housing. So we need another form with another objective. This system needs to be flexible enough to accommodate those possible scenarios.

Consistent. An important rule of data management is to be consistent actually. So when we first start building the system we need to store and organize information. So in order to continue using the system and not have to harmonize at the end of the day. Some formatting conventions are arbitrary and only matter to the extent that they're applied consistently. Similar to choosing on which side of the street you drive practically.

00:27:16 Steps in setting up an information management system

What are the steps now in setting up an information management system? I have grouped the steps in three components. Always we start with the planning. So what we found, what inputs we have during the plan, we have frequently the monitoring and evaluation planning documents. Here we have the data and reports requirements. It could be the case that during planning we have specific policies and procedures to follow, whether that indicates whether our project requires data privacy impact assessment, the dataset agreements, who has access in which type of data. We may find ourselves with other guidelines, Sphere book guidelines. If we are offering protection case management activities, the protection-related guidelines. And all this information of the planning leads actually to a specific output which is system requirements and our data model.

Now what we need to do on the next phase, on the design, is take those system requirements, those data flows, those roles and responsibilities, the data model, and transform it at the end of the day into a database. So here we start designing the actual forms by which we're going to collect information. We design our reports and we have all those naming conventions, how I'm going to name specific fields and whether this field that I'm using here is the same field that I'm using elsewhere or files connection. If also we use a system that we can attach and bring documents. The output of this phase is the database design and the reports design.

And then finally we find ourselves into the launching procedure. So here we have the system built in place. But of course, as I mentioned, the information management system operates within the broader context of people, technology, and processes. So in order to be able to launch quickly, efficiently, we need capacity building and available documentation. And this comes down to also processes and people at the end of the day. And we need also to be able to monitor that system and disseminate information, get feedback by whoever actually is using that system. So here the main output is any available documentation. As available documentation, frequently we have presentation templates, the manuals, or any other relevant materials.

00:33:01 Common challenges

What are some common challenges now that we have on a specific environment? And I will frame those challenges around people, processes, and tools.

Challenge number one: here we find ourselves with an influx of new staff and staff turnover. So as we may understand, we have some capacity building implications. So we find ourselves in a position that we need to get the new staff up to speed to retain or easily find knowledge. When all staff leaves practically here it helps when we have standard templates for data collection analysis, specific guidelines, specific SOPs that we can take from the region, we can take from the HQ and implement inside the country. Use of the resources as I mentioned, this is more efficient. Especially the case that an organization uses a specific set of tools in order to collect information. So the organization has readily available manuals and readily available templates during the process.

We find frequently ourselves in a position that we lack reliable, credible, or up-to-date secondary information that we use in that context. Of course, we have a short time to collect and analyze the information. Information sensitivity, data protection security is always important; is a bit more important I might say in a fragile context. In a context that actually if the beneficiary data that are being found in the wrong hands it can cause harm, it can cause harm to the person that they try to assist. It can cause also harm at a later stage. Organizational reputation for example. Here as a tip I find useful the use of common operational data sets that I have a link is the one maintained by OCHA. Use systems as I mentioned that has established capacity and also those systems they need to be flexible and simple to use in order to make any adjustments as required.

And of course when you work with different stakeholders, especially people, they work in clusters with different patterns. Usually common information management system like you don't want to enter in a position to integrate between different systems, it's time-consuming. But you want to be in position to deploy a system that can be readily available and needs to be used at the cluster level at the organization that works with a different partners level.

When it comes to tools we need to be aware that we need a tool that enforces data quality because we need to reduce time from data cleaning especially always but especially when we don't have sufficient resources in place. And of course data protection security is crucial. So we need also a system that allows us to implement those roles, implement those permissions. Specific user will have access to specific data collection forms. They need to be able to view specific data because they come from number X and they are not in a position or they shouldn't be in a position to view the data of a partner.

00:36:32 Case study: ActivityInfo example

Let's see an example of a case study that I had designed for the specific webinar. As I mentioned we have during the design three big steps: the planning, the design, and the launch of the system. So let's see an example. I mentioned earlier that frequently we use monitoring and evaluation documents and at the end of the day we gather the data requirements and we compile everything at the data model that we want to have at the end of this process.

So if we see an example and let's assume in my scenario that we work in an emergency response and we are an organization that decides to focus on a CASH assistance intervention in order to satisfy the immediate needs of target population. So how a very simple example would look like: we start with the result framework. We always have a long-term goal that we will achieve years from now. We have a strategic objective that what we want to achieve is that this target population meets their needs. How are they going to meet the needs? Because they're going to use cash assistance. How they're going to use CASH assistance? Because I'm going to start distribution to CASH assistance of the target population.

It can be in the long run that I add another strategic objective. Example given, at a later stage I may need to create linkages in the labor market in order for these people to be able to live independently and have their own salary. So I may add a strategic objective referring to that. And an example that I thought of in terms of feedback mechanism is when we include the cross-cutting intermediate result that the target population has timely access and response via the feedback mechanism. So this, when I include this cross-cutting statements, it helps me also formulate an indicator, something that for me it will be useful to examine and analyze my data around.

And then a result framework becomes an M&E plan. So let's focus on the down level because this is the most important for the emergency response. As I mentioned, intermediate result, the output level, the activity levels, and the cross-cutting intermediate result. So here I have used one to two indicators. So remember to keep it simple. Generally, a rule of thumb on the indicators is 1 to 3. Never use more than 3 because in this case then it will be too complex to manage the M&E plan and you don't want such a complexity, especially in those first months of implementation.

Indicators on the output level could be total of number of people that they have received CASH information, how many people they have registered in order to receive cash, how many people I have assessed in that specific process. In terms of the feedback, it could be the percentage of feedback that was addressed within three working days. This goes according to the SOPs and here I have all the information needed on the indicator level. How I'm going to collect, how I'm going to analyze, how I'm going to calculate, how I'm going to use that information and whether at the end of the day I will have any disaggregation.

Now this document, and of course it comes with other documents frequently found in organization—the SOPs of the feedback mechanism that may have on a regional level, the SOPs of data protection that I may also have—they're being transformed into a data model. This is my last step. So practically they're being transformed all those different information of my context combination with standard operating procedures into the different roles that I may have within a specific project. The M&E plan is useful because it gives me the specific data that I have, whether it's data that are subject to data collection, registration, surveys, focus group discussions, annual service or whether the standard data sets, standard data sets that I want to use for this project and for the next projects as well, my partners, the location, the different location levels.

Then we transform this information so we have the different roles and the different data sets into flows. Who is doing what via which data collection flow within a system. At the end of the day, we want to end up with a data model that illustrates all the details of that system when it comes to the beneficiary registration. What information I'm collecting on the beneficiary registration, the specific fields, but the beneficiary registration is associated with the CASH distributions and I know that one beneficiary will receive many CASH distributions and will use CASH multiple times. One beneficiary can be the case that I have many cost monitoring distribution service for the specific beneficiary and I have also a data collection form for the feedback mechanism that I want to keep separate because I acknowledge, as I mentioned earlier, I acknowledge the fact that I'm going to receive maybe feedback in such a context of participants and non-participants and both is equally important.

Then the next phase is taking practically taking the data model. This is what they crucially mean combining it in the back of my head with a different data flows. Because I need to create a system at the end of the day that will include all the data collection forms and will have all the different roles, permissions and responsibilities. So then I move into the next stage which is a database design and how database would look like. As I mentioned, we need to organize the data. So in our system we use the folders to organize the different data collection forms.

If we separate the data sets, the main categorization is between the standard operating data sets that we call internally reference data and the main data collection forms. So we have the different data collection forms that we have our major registrations, specific registrations, registrations and those are associated as I mentioned with the CASH distribution, the assessment, the cash usage. We may have all the different services, the service that support also as I mentioned intermediate level result monitoring but also collecting feedback via the focus group discussions. But it could be the case that they support the real-time evaluation also because I have included specific tools in order to be able to combine the monitoring data with the real-time valuation and of course the feedback mechanisms.

So as I mentioned here we have the feedback form which is separated from the beneficial registration because exactly I acknowledge that I am receiving from participants and non-participants and it could be the case they receive multiple times. So perform the first registration. But I have cases that this person has given me two times specific feedback and how feedback can look like usually the most common field that we use in those forms is how did I receive, the channel. The category of the specific feedback information request or whether it was something out of scope. All this information that are being defined using the standard operating procedures document. And of course when I close the specific record and how many days it took me. Because if you recall I have also an indicator to count the percentage that they were close to in less than three days.

And then during that phase I tend also to design all the different reports. So we have all the different reports and because we want them to be public to be available to different stakeholders. Now so to this end I have created specific report that has a specific indicators that I monitor on the lower level per month because this is what I was intended to do. It can be per week also then I may find myself on having the cumulative progress because I will not be in a position to monitor my progress against the target. But also it could be the case that I made disaggregations. I have the other benefits registration. I need to see how many people they are present age group across different administrative areas.

And then moving to step three we find ourselves in a place that we need templates, we need manuals, we need to perform those capacity building sessions easily in a way. So one example that I thought is to provide you with based on some internal templates that we have in order to provide you with the fact that if you use a standard tool within the organization you can take a standard template and then customize it. So usually few things that remain common in those implementations. The overview of the system is always common. How I log in in the specific system is always common. How I add records is always common. It differs on the nitty gritty of the specific data collection. For the things that they differ majorly is the setup of the broader database. But the main instructions they remain the same. That is why it's easier to deploy fast.

00:48:43 Q&A session

Great. Thank you so much Eliza. And thank you everyone for being so interactive in the chat. We already have some questions just to share that we will share the recording with everyone who has registered to this webinar and it will also be available on our web website tomorrow or by the end of the week. And yeah, Eliza, back to you.

Okay, let me see the questions. So, is there a way to extract the data model from a database or should we create it manually? Great question. So for the moment we have plans in the future to make it non-manually extracted. For the moment we design it in external systems that we use, we visualize it, but we have it in our minds because we know what is useful and we know what is an existing need in the information management world both for emergency and emergency programs. So the short answer is that for the moment we create it and all our customers they create it in some visual ways, we use Miro internally. Other people they create it in Microsoft, Visio, et cetera in order to work with it and visualize it. But I agree with you that it's a nice need that we have identified already.

Can you share the manual template and users? So I will need to consult internally because this is the plate that I had created. So I will consult internally and I will let you know because it's an internal document. So it was useful for the purpose of the specific webinar because I wanted to give you something that would be meaningful and an example is always meaningful. We can get back to you.

What are the key challenges in Collective at 5W Matrix by UN agencies and single reporting system? How much these data collections are variable, in your opinion? So the 5W. I mean, the 5W is a system that is being used across all the UN agencies. So from the information management perspective, and to the extent that I was involved in the design of such a system, it is viable or I haven't heard any serious complaints of having them in a single reporting system. On the contrary, I would say that having them in a single reporting system facilitates data collection and the data use. From my perspective and to the extent that I have seen the implementation of those tools people they have reported back, that is streamlines data collection and data use. At the end of the day.

Can you link other data collection tools? Example given go collect activities. So what happens with Kobo? Kobo as a purpose shares the same purpose with ActivityInfo. So the short answer is that we don't have an integration because there are two systems that says they serve the same purpose. We have integration with systems that they serve complementary purpose. Example given ArcGIS, QGIS for geographical analysis, R for statistical analysis or POWER BI for more advanced data visualization analysis. With Kobo though because exactly we serve, we have commonalities and objectives that we serve. We have made it possible to integrate easily the data in case that you, the organization or the project wants to use an information management system a database in order to operate rather than... Which is great of course for its purpose on this survey collection. But if you want this survey example data collection or any other forms to exist in a common system with responsibility we have made it possible to transfer from Kobo. The keyword is the XLS form.

What are the most difficult challenges in building and operating an emergency M&E system? So as I mentioned, I think some of those M&E challenges coincide with information management system challenges. So if you take it a step back, the staff turnover creates also an issue. So right there, if you have specific guidelines at organizational level facilitates the process. Number two, it creates an issue this staff turnover. Because people they make something then they leave and then the next person line should continue that something. This creates a silence because frequently we found ourselves that we don't have transparency. So the initial documents that they were created either this was the system requirement documents into the data flows do not exist somewhere in a way that can provide all the information. And that is why I showed you the example because from my experience maintaining those system requirements, the data model on very specific and detailed format facilitates the whole process in reality.

Another thing, the fact that frequently we don't have an information management system in place timely creates issues of the monitoring and evaluation because you find yourselves in a position that different people they're using different formats those Excel files in different locations. So that is why the more as organization you put some effort on thinking through. I use X tool for information management how I can facilitate the adoption in an occasional emergency or in a new project that starts by X country. What do I need and think it holistically. I need instructions, I need standard operating procedures. I need to have a pool of people they're aware capacity and they can easily capacitate other people in their permanent staff of my organization. So think it holistically in order to address those challenges. Because the delay on setting up the information management system affects also the monitoring and evaluation system and it affects on the fact that I don't get quality data.

Someone is asking whether they can link all the data to single report format. Yes, as long as the data collection forms are linked. So this is the nice thing with the relational data model example. If I have beneficial distribution collecting at point A in time and then I collect an assessment in point B or cast distribution in point C sitting in different data collection forms because different people they have access they've been collecting different points in time. They have different objective. At the end of the day the relation data model assumes that you create those relation between those tables. So this information can be used and be brought either summarized, depend on the relationship or if I need. In these cash distributions, the beneficial information also can be brought. So yes, they can be turned in a way to a single report format, depending on how, of course, the report is being formulated. Because each report has specific needs in terms of data restructured in that sense.