00:00:01 Introduction and agenda

Let's start with what we are going to see today. First, we will set the scene; we will introduce the key definition of what is actually MHPSS in order to be on the same page. We will focus on the monitoring and evaluation framework for these kinds of interventions. We will draw attention to ethical considerations around monitoring and evaluation, such as consent, data collection, and data analysis.

Then we will move from theory to implementation, so we will try to make the case for how we can actually bridge the gaps between monitoring and evaluation and case management for these kinds of interventions and how this can be implemented with the use of information management system technology. I'll showcase an example based on ActivityInfo, which is the software that I introduced in the beginning.

Before we start, let's make a short poll in order to make this webinar a bit more interactive. In terms of the information management system that you're using, the majority use separate, fragmented tools. Manual systems come second, and the third one is 'not applicable'. In terms of satisfaction, the majority are somewhat satisfied.

00:03:03 Understanding MHPSS

Let's introduce MHPSS. It is used to describe any type of local or outside support that tends to protect or promote the social well-being and/or prevent or treat mental disorders. This is according to IASC guidelines in 2007, which was a great resource for the purpose of this webinar, both in terms of the monitoring and evaluation framework, but also in terms of examples that we will see moving forward.

The characteristic of MHPSS interventions is that they are multi-sectoral and cross-cutting. They are frequently embedded within health, protection, education, nutrition, WASH, shelter, camp coordination, management, food security, and livelihoods.

Some key considerations that we need to have in our minds before we proceed with monitoring and evaluation include the fact that effective MHPSS programming requires close coordination among diverse actors across multiple sectors. This ensures that the full range of complementary MHPSS activities and services are provided in line with global guidelines. It is important to assess MHPSS needs and resources in order to guide planning of more detailed and context-specific program activities.

In the design phase, information is gathered through specific coordination mechanisms and assessments. This information should be used to identify existing resources, prioritize needs, and select or adapt program activities and actions in collaboration with key stakeholders. The M&E framework should be developed as part of the initial program design. Where MHPSS is being incorporated into existing program activities, the M&E frameworks should be updated to include those components during implementation.

Another crucial component is MHPSS orientation and advocacy, which can influence different humanitarian or development actors to take specific considerations into account. MHPSS orientation and advocacy can help decision-makers to recognize the value of MHPSS, and this in turn can lead to increased resources, including funding allocation. Something that is very important, specifically in these kinds of projects that address sensitive topics, is capacity building and staff care in order to guarantee efficient design and implementation.

00:06:23 From case management to structured insights

How can we move from case management to structured and clear insights? Case management reality is emergent. It cannot be explained from the behavior of individual components, but it rather emerges from the interactions with individual components. It is not only linear, due to the ways in which interactions between multiple components of the system accumulate. Small changes in the behavior of individual components may result in disproportionate effects on the global state of a system.

It is also adaptive; those individual components of the system can change their behavior to adapt to changes in the behavior of other components. It is a complex reality that we are living in, specifically the case management components. Thus, the monitoring and evaluation framework and the information management system help us organize and define the data that we will collect. Ultimately, we can construct them to gain clear insights.

00:07:27 Monitoring and evaluation basics

Monitoring and evaluation is necessary to assess whether or not a program, project, or intervention is achieving its desired results. It must be built into the activities from the very beginning. In this context, monitoring refers to the visits, observations, or questions we ask while a program is being implemented, to see if it's progressing as expected. For example, one case you should consider is if the program is doing no harm. Monitoring can help to assess whether, indeed, this is the case.

Similarly, evaluation refers to examining a program at the beginning, middle, or after it has been implemented to see if it has achieved the desired results. For example, a project might aim to reduce symptoms among people with specific mental health conditions. The severity of symptoms, along with other indicators in the project, such as the number of personnel involved, risk and protective factors, could be monitored throughout the life of the project. The severity of symptoms may also be evaluated when people are first seen by service providers (baseline), at points during project implementation (midline), and at the end of the project (endline evaluation).

For the purposes of this webinar, we follow the IASC guidelines. It's worth going into the details of what the establishment of a monitoring and evaluation framework means. First, we need to establish the project objectives. This is known as a results framework and helps us establish the project logic. The next step is to choose the indicators and respective measurement methods. It is strongly recommended to use quantitative and qualitative measurements.

Data collection plans help us define the procedures of data collection: who, how, when, where. It supports data protection with the creation of standard operating procedures for the collection. This information—objective statements, indicators, measurement methods, and data collection plans—are frequently consolidated in monitoring and evaluation plans. At the end, we report back results to various stakeholders.

00:10:15 The common framework and definitions

The purpose of a common framework is to encourage the use of a select number of outcomes and indicators, and the use of common tools to build the MHPSS evidence base, and better inform those working in this area about important goals and impact. Of course, each MHPSS program is unique. What M&E specific programs use and how they are implemented will depend on the context, assessed needs, the experience and capacity of implementing organizations, and local resources.

Let's establish some key definitions. The overall goal is the specific end result desired or expected to occur as a consequence of relevant project outcomes being achieved. A portfolio of multiple programs may be necessary to achieve an overall goal, such as reducing suffering in a target area. The outcomes are the changes that occur as a consequence of specific project activities. Results at this level are commonly referred to as project outcomes. Activities are the actual work that's being implemented. It is the level of an activity often referred to as outputs.

00:12:08 Goals and indicators

The proposed goal, which is reducing suffering and improving mental health and psychosocial well-being, comprises two important elements. First, the goal is to reduce suffering, aligned with the humanitarian principle that action should be taken to prevent or alleviate human suffering arising out of a disaster or conflict. Second, the goal focuses on MHPSS by asserting the aim is to improve people's mental health and psychosocial well-being.

Under the goal, there are recommended indicators in several areas. Functioning refers to the ability to carry out essential activities for daily living. Subjective well-being includes feeling calm, safe, capable, interested, and not overly helpless, depressed, or angry. Other indicators include the extent of prolonged, disabling distress, and indicators of MHPSS results, such as the ability of people to cope with problems. Social behavior indicators might track helping others versus aggressive behavior. Social connectedness refers to the quality and number of connections an individual has.

The outcomes are grouped into two categories: community-focused and person-focused. Community-focused outcomes center on social considerations for safety, human rights, dignity, and basic needs. Examples include ensuring emergency responses do not cause harm, people are safe and protected, and social structures promote well-being. Person-focused outcomes center on individuals and families requiring targeted assistance. Examples include community and family support for people with mental health problems, and the use of appropriate focused care.

For example, regarding community and family support, the rationale is that these are typically the first support measures people receive. Indicators might include the level of social capital of individuals post-intervention, or the number of people reporting receiving adequate support from family members. For the outcome regarding appropriate focused care, indicators might include levels of satisfaction regarding care received, and the number of women, men, girls, and boys who receive focused psychosocial care.

00:17:58 Measurement methods

Means of verification (MoV) are the quantitative and qualitative tools used to measure the indicators. For example, if the indicator is the number of individuals reporting reductions in symptoms, the relevant measurement tool is a questionnaire with relevant questions.

We select the appropriate means of verification based on six criteria. First is relevance: the match between the assessment approach and what needs to be measured. Second is accessibility: whether the tool is freely available and in the required language. Third is feasibility: how easy or difficult it is to administer the tool in terms of time and resources. Fourth is acceptability: the attitude of the users and whether the tool causes harm or distress. Fifth is reliability: the extent to which a tool produces stable, consistent results across time. Sixth is validity: the extent to which a tool measures what it is intended to measure for a particular setting.

It is recommended that a mix of quantitative and qualitative means of verification is used. Age range is a crucial factor. You can always opt for the development of participatory indicators and MoVs. We need to consider the context, language, and culture prior to adopting an existing M&E tool. We need to pilot and adapt it prior to implementation. If we use an existing M&E tool, we need to provide training to staff administering it.

00:21:56 Ethical considerations

Applying ethical principles to all aspects of monitoring and evaluation is important in avoiding potentially risky practices and keeping people involved safe. The most important tasks include having strong data collection plans and procedures in place. These plans indicate the process of the collection: the purpose, what is being measured, when, how, and by whom. It also covers the required procedures for protecting and ensuring the confidentiality of respondents' information.

We need to train collectors according to established plans and be clear about the importance of obtaining informed consent. We need a code of conduct for collectors regarding expected behaviors. We must ensure the selection of appropriately qualified data collectors and consider how their characteristics might influence responses. We must also plan for how data will be recorded and how issues like cleaning missing data will be managed.

Data disaggregation is an important factor. Data should be inclusive for particular groups who have special needs. Indicators must collect data that are disaggregated, including but not limited to gender, age, and vulnerability types. Care should be taken not to cause any harm, such as reporting small groups in a way that makes them identifiable.

Sampling refers to the individuals or groups from the target population included in the data collection. Probability or representative sampling means the sample is randomly selected and representative of the target population. Non-probabilistic sampling, such as selecting people because they are conveniently available, implies the sample is not representative. Sampling decisions have ethical considerations; we want to confirm the approach is appropriate and that we only collect "need to know" data.

Informed consent is the permission that people give before agreeing to share information. As a minimum, it should include names and roles of organizations, the purpose of collection, how it will be used, procedures, potential risks and benefits, the right to withdraw, and how identity will be protected. If they are a specifically vulnerable group, additional care should be taken.

00:28:45 From theory to implementation

Data collected that are not used should not have been collected in the first place. To guarantee timely use and relevant lessons learned, we need to operationalize the monitoring and evaluation framework by using a common information management system. This combination guarantees that information flow can be implemented, reported, and learned from timely.

The information management system supports the data process from production to consumption. Key components include data collection (high quality), data organization (reliability), data processing (analysis), and data use (actionable information for decision making). We need to operationalize permissions for data protection, perform analysis, and share results.

During project planning, we decide on MHPSS case management activities, milestones, requirements across stakeholders, and learning requirements. We determine how data requirements translate into data collection tools. During implementation, we identify how data is transformed into actionable information and define the data collection flow.

Typically, multi-sector approaches need to be reported to different stakeholders, such as technical working groups. Data collected on activities needs to be cleaned, analyzed, and transformed into indicator calculations. Using a common information management system facilitates this. "Common" means creating a system that different partners have access to, with permissions defined by specific project procedures. This increases the importance of data protection and documenting the data collection plan.

00:35:16 Case study and data modeling

Let's look at a case study of a multi-stakeholder project. The goal is for people living with mental health conditions to experience improved mental health. Outcome 1 involves nurses at primary healthcare facilities identifying and supporting individuals. Output 1 is nurses receiving training, and Output 2 is the provision of psychiatric medication support. Indicators include the number of individuals reporting a reduction in symptoms, increased availability of essential medicines, and the number of people receiving clinical management.

To model this in an information management system, we must combine the M&E framework with the case management process. We have the patient for whom we collect basic demographic data. For this patient, we collect other data points at different times, such as healthcare records (e.g., WHO assessment schedule), referrals, and case closure. Simultaneously, we need to monitor healthcare facilities regarding essential medicines and trained workers.

By modeling the data this way—registering beneficiary persons and linking multiple healthcare records to them—we avoid duplication. We collect demographic data only once. Data protection is crucial; we restrict access based on roles. For example, a health worker might only have access to beneficiary forms they are assigned to and can only view, add, or edit records. A supervisor might have access to all records for their assigned health workers and have the ability to delete records.

00:40:07 ActivityInfo demonstration

In ActivityInfo, we can organize this into folders. We have case management forms and beneficiary forms. We can navigate participants and their associated records. The system guides us to the next components that need to be filled in. We also monitor health facilities in a separate folder, tracking the availability of medicines, which can be updated easily.

The system allows us to identify partner access roles. For example, logging in as a health worker, I can only see the folder and members assigned to me. I can edit records to update information but cannot delete them.

Regarding data use, different levels of access are required. Health workers need day-to-day access to specific information. Project managers need monthly reports for technical working groups. We can design reports that aggregate data based on indicators, such as the number of persons receiving clinical management or the number of facilities with trained staff. Visuals like monthly registration per facility and gender breakdowns facilitate reporting and discussion.

If using a common framework across multiple projects, the lead organization can create templates. These templates can be modified for each project while keeping minimum requirements for standard reporting. Standard lists (reference data) like partners, locations, and vulnerability types can be shared.

00:47:44 Key messages

Better data lead to better decisions and better MHPSS outcomes. Monitoring and evaluation isn't just paperwork; it's how we check that support is helping and not causing harm. Impact is measured using indicators (what we look for) and means of verification (how we measure it). Good measurement tools must be relevant, culturally appropriate, safe, ethical, and respectful.

Numbers and personal experience matter, so collect quantitative and qualitative data together. Personal data must be protected, and data should only be collected if really needed. An information management system helps turn raw data into useful insights, supports secure data collection, clear roles, and timely reporting. A shared system allows multiple organizations to work together while protecting sensitive information.

00:49:53 Questions and answers

Is the database linked to any mobile application? Yes, it is linked. It's available both through the mobile application or web interface, offline and online.

How can MHPSS be integrated in economic empowerment and education projects? There is guidance in IASC specific projects, but I will need to go back to my resources to provide a completely consistent response. We will add some Q&A on the webinar page.

Is there a MEAL framework template? Usually, you use the template that is appropriate for monitoring and evaluation within your organization. If you search for guidelines on M&E plans, this applies to MHPSS interventions. For specific outcomes, indicators, and measurements, the IASC guidelines are a useful resource.

How can we adapt the IM system design for MHPSS to other sectors? The adaptation depends on the monitoring and evaluation framework. The first step is to go back to the M&E plan with indicator statements and measurements. This information is then modeled visually into a data model and data flows, which is then translated into the system.

How do we ensure our job is safe with the rise of AI? I am a bit skeptical that AI can do most of the manual work yet. It can do some parts, but with caution. I wouldn't say that in the near future AI is substituting monitoring and evaluation.

Do we have baselines for standard indicators? Baseline and target values are usually based on an assessment or a specific project.

Does the database have a dictionary? In information management, common terminology is used. In ActivityInfo, we have specific terminology that is publicly available.