00:00:04 Introduction

Thanks, Faiz. Really excited to be here with Talib from Syria. I started my career at UNICEF working in the emergency section in Congo, and that is where ActivityInfo came from. So it is always really exciting to me to see ActivityInfo continue to be used and grow within UNICEF. I particularly enjoyed working with Talib in Syria. A lot of the development that we did last year and the year before on calculated measures was really driven by Talib and his team in Syria and the indicators that they needed to calculate.

Thank you, Alex and Faiz. Good evening, colleagues. This is Talib from the Syria country office in UNICEF. Thank you, Alex, for this opportunity to showcase the UNICEF Syria collaboration with ActivityInfo on how to generate evidence and reports to inform the decision-making process for the strategic level and implementation level. In this webinar, we will talk mainly about the results-based management utilization in Syria by using ActivityInfo connected with data aspects and also how we are using ActivityInfo in terms of pre and post assessments at the program level.

00:01:38 The 4Ws workflow and data architecture

First of all, I will share my screen to show you the workflow of the data we are working with in Syria. I will focus mainly on the 4Ws because, as you know, Syria is under the humanitarian response plan context, which is led by OCHA. So we are using the 4Ws to measure our progress towards planned targets. The 4Ws is mainly focused on collecting information from implementing partners that have active PDs with UNICEF on the ground through specific templates in order to produce output and outcome reports and indicators.

As you can see on the screen, our 4Ws skeleton is built on having specific databases on ActivityInfo. Each database is dedicated to an implementing partner who has an active program document signed between UNICEF and these implementing partners by selecting specific standard indicators to implement these activities in some locations. This will lead us to have a form of 4Ws. In the second milestone, we have IP databases. The implementing partners are collecting the 4Ws from the static centers and mobile teams supported by UNICEF and consolidated in the HQ of the implementing partners. The implementing partners can be either national NGOs or INGOs.

This will be consolidated in one database on ActivityInfo. The database is built based on relevant tools and data validations that will accept records only that match the signed program document. This will give us a first layer of automated data cleaning. After that, there will be another layer of data cleaning that will be led by the information management officers in UNICEF intersections and in the PME before hosting this information in the last milestone in the data entry stage, which is the ActivityInfo master database we call the 4Ws. Once this clean data is hosted in the master database, there will be another layer of calculations, fields, and tables applied in order to deduce the reach of UNICEF interventions on the ground at output level and also at outcome level. This leads us to feed the standard products of UNICEF, such as situational reports, CSI, RAM, SitRep, Humanitarian Action for Children, and the Humanitarian Response Plan commitment under the Core Commitment for Children.

00:05:13 Database setup and reference data

In Syria, we are depending on ActivityInfo for different reporting levels starting from outcome, outputs, input activities, and we have some project monitoring databases. In Syria, we have twenty-four active databases. Most of them are run by the PME section while the remaining are run by the programs at the project level. To highlight the capacity of ActivityInfo in terms of hosting records, in 2023, we hosted more than 500,000 records through 4Ws. In 2024, up to May, we hosted clean 4Ws that reached 134,000 records.

Regarding how we are reporting the 4Ws, at the beginning, we have something called reference data. The reference data contains the location. In Syria, we have 8,500 locations supported with latitude, longitude, and P-codes. We also have the input activities that we use to report through 4Ws, specific indicators, and the partner list. Here we are listing all the partners that have active agreements with UNICEF. In this case, our baseline to start the reporting following the bottom-up approach is ready with this reference data.

After that, we come up with the reporting module, which we call the 2024 programs imported 4Ws. Here, we are reporting the interventions that took place following the bottom-up approach. Geographically, our report is limited to report the interventions started with admin 4, which is communities, and sometimes the admin 5, which is neighborhood. In terms of UNICEF programming, we are reporting versus input activities and those activities together will frame the indicator supported with some calculation methods to reflect the unique beneficiaries reached with zero duplication.

00:08:19 Data entry and validation

As you can see, we have two modalities of recording: either record-by-record or bulk import. This form is built using the reference data in terms of specific locations and specific partners supported with searchable and dynamic drop lists. For the data entry level, we are seizing the human errors. We have the activities, the locations, the address, and also the modalities of reporting the type of beneficiaries—if they are host communities, internally displaced, or returnees. We also have a question about disabilities, if this intervention is reaching people with disabilities or not, and then the disaggregation of the beneficiaries using the GAM methodology, which is age and gender.

We connected the activity targeted audience with the beneficiary. For example, if this activity targets only adults, ActivityInfo will not host figures that target children and vice versa. This allows us to have much cleaner data in an automated way rather than following human inputs for data cleaning. Also, because UNICEF is working through implementing partners, each implementing partner has its own signed PD. The signed PD has specific indicators to work in and specific locations. So here we have another data validation layer. If I am recording a record to a specific implementing partner, ActivityInfo will immediately connect this record with the signed program document and will allow me only to put inputs related to this program document geographically and programmatic area of responsibility. This built-in data validation helped us to host as much clean data as possible in the first layer of the data entry.

00:11:13 Calculated fields and indicators

Now I will take you to another layer of reporting. In the 4Ws database, we have the reference data and the raw data. This leads us to have calculated fields and measures to come up with output indicator progress and outcome progress. Here we cluster the activities that feed specific indicators. This indicator is supported by syntax using DAX methodology to apply the max or sum of a group of activities to provide figures that report this indicator.

For example, for the indicator "number of children/women accessing primary health care," we can see the syntax built through ActivityInfo and how we are grouping the activities in order to apply the calculation method. We are telling ActivityInfo to use the list of activities from reference data and to take the report from the community level, taking into consideration the group of people with disabilities and the modalities. Then I give the instruction to consider only two indicators with specific codes. ActivityInfo will look at the raw data of the 4Ws, look at these two activity codes, and apply this calculation method, taking the sum of these two activities to consider it as an indicator reach. In other cases, we have a group of activities where we take the highest reach of three or four activities.

ActivityInfo also supports us in building a pivot of this indicator. Here we are seeing a list of the indicators, and in the columns, we have the field office and the group. If we delete the field office, the report structure will reflect the national reach of these indicators. I can filter by governorate, by community, by people with disabilities, etc. The pivot will apply all these calculation methods of these indicators at those levels.

00:14:15 Visualization with Power BI

Through the token and API reports generation from ActivityInfo, we can connect all these reports into Power BI in order to build up a dashboard. All the reports you have seen are connected with Power BI in a dashboard called "Program Results 4Ws." For example, all our work is built on the Humanitarian Action for Children, which is the annual plan for UNICEF. In yellow, we are reflecting the targets we are aiming to reach. In blue, we are reflecting the progress and interventions provided through UNICEF and its implementing partners. All the blue figures are generated from ActivityInfo.

We also have a table of the status per each program—if it is on track, off track, etc. The progress itself is extracted from ActivityInfo, based on the reports, outcomes, and outputs we were talking about. Whenever we are feeding ActivityInfo with 4Ws, the reports will be updated automatically and reflected in the dashboard. This gives us the easiest way to build decisions in terms of our response, to scale up or scale down, or to see which areas we are focusing on. We can also produce maps from ActivityInfo connected with Power BI and have specific analysis beyond the indicators.

00:17:40 Pre and post assessments for MHPSS

Now I will move to another area, which is the pre and post assessment. We supported our child protection section in Syria to run pre and post assessments for the MHPSS (Mental Health and Psychosocial Support) program. First of all, we built our databases using the reference data of the locations. We mapped all locations where UNICEF is implementing the MHPSS program. Then we mapped the partners that are using these key interventions. Also, we identified or mapped the enumerators who will meet the beneficiaries in order to run the pre and post assessments.

In order to keep the data protected and also to link accurately the profiles of the beneficiaries between the pre and post, we provided each enumerator with a username and password and gave them access to the pre-assessment. The partner is granted with conditional view or access on ActivityInfo. If I am working with UNICEF as an implementing partner, ActivityInfo immediately takes me to the space of UNICEF. I have to provide the username and password. When I enter the password, it will open the questionnaire and start the interview with the beneficiaries.

At the end, I have to give a child profile or survey profile code, like 123, and I should keep it with me. As a pre-assessment, I have done this interview. Then the beneficiary took the interventions through sixteen psychosocial support sessions. Then I have to redo the same assessment, which we call a post-assessment, to the same beneficiaries. To link between the pre and post, I go to the post-assessment, add a record, and ActivityInfo will automatically generate validation. I have to inject the partner name, the username, and the pre-assessment survey number. When I provide the survey number, it will automatically provide me with the results of the pre-assessment, and then I continue the post-assessment result. This will also be visualized and generate analysis through Power BI as an interactive dashboard.

00:21:28 Q&A: Managing partner data

Alex: You presented two use cases here. One is collecting information from implementing partners, and the second is collecting information directly from beneficiaries. Is it correct that you have a staging database for partners' reports before you put that into your master 4W database?

Talib: Yes, because this is a new strategy. As a rollout to build the capacity of the partners, we have a specific database for each implementing partner for the time being for 2024. We are in a stage where we are connecting ActivityInfo with our eTools, where we sign our program documents. For the time being, each implementing partner has its own database. Next year, there will be no staging; they can report directly on the master database, but there will be conditions where the data will be locked to be cleaned.

Alex: How many partners are involved in this reporting?

Talib: I would say more than fifty. They don't report at the indicator level; they report at the input activity level. Each activity contributes to a specific indicator, and there will be a calculation method applied based on clusters of activities to provide the results of the indicator.

Alex: So because you are working with fifty different partners, each of those partners has their own tracking system, and this is the means of bringing that information from all of those different systems together in one place so that UNICEF can see it.

Talib: The program document manager has a work plan with the implementing partners, and they agree on a plan of implementation. For example, we have to roll out vaccination to reach 200,000 children in four governorates. There is a monitoring visit that verifies this vaccination implementation. Once it is verified, on a monthly basis, the implementing partner reports this implementation through the 4Ws.

00:30:51 Q&A: Quality assurance and system integration

Tetiana: You mentioned that there are three levels of quality assurance, and you explained two of them. I am curious to know the third one. Also, how do you link ActivityInfo with eTools? It is often a burden for partners to report in different places.

Talib: Regarding the third layer of data cleaning, when the programmatic verification takes place, we move to the third layer of quality assurance run by the PME. We have the "people in need," population census, severity scale, and the target per each program document. We run quarterly data cleaning. For example, if in a specific community we reach more than the population there, we have to do some capping. Sometimes we are reaching people in low severity areas, which we don't consider as people in need. In this case, through high-frequency reporting, we can direct the program manager to scale up, scale down, or revise the implementation.

So the first layer is technical/automated, synchronizing with the program document structure. The second one is programmatic, verifying the people reached. The third is technical/statistical in order to be under the hat of HNO and HRP.

Regarding eTools, we have built the database of ActivityInfo to talk to the program document structure. Whenever a new partner signs a new PD, we immediately register this partner in ActivityInfo and select the indicators that the partner will work through. This grants them access only to those outputs and locations.

00:41:40 Q&A: Access rights and target management

Rukati: Is ActivityInfo section-led or centralized? Can every section create or contribute to the form, or is it done at the PME level only? Also, regarding the dashboard, where do the targets come from? Do you have a separate file for targets?

Talib: In Syria, we are following an agile approach. As PME, we are the database administrators for ActivityInfo, so we are designing all the forms and reports. At the section level, each section has its own access for its space. For example, an information management officer in child protection can see only the information related to child protection. In terms of implementing partners, for the time being, they have access to their own databases only, linked to the program document structure. This is granted through the user management agility that ActivityInfo provides.

Regarding targets, the results are imported from 4Ws. In terms of targets, in Syria, I have a separate spreadsheet, but in ActivityInfo, you can have a specific report for the target and can be interlinked together. This year, I used Excel to build up the targets and connected it with a pivot table in Power BI to connect the program's targets.

00:49:28 Q&A: Qualitative data and future features

Amin: I have a question about qualitative data. How do we collect it with output indicators?

Talib: I would say 90% to 95% of our data is quantitative, but we have some qualitative indicators. We cannot use qualitative analysis based on free text in ActivityInfo currently for analyzing, but we can go with scaling—answers like scaling from one to five or "good," "very good," etc. Consolidating results from focus group discussions is something we have to think about.

Alex: Quantitative data is always easier to analyze, but you do miss a lot by not including qualitative data. Some ways I have seen ActivityInfo used is for collection and indexing. You can code qualitative data. Ultimately, the most effective way on a scale is to collect data and then look at the categories you are seeing and assign codes.

Amin: Does ActivityInfo have a mobile app?

Talib: Yes, we are using it. For the pre and post assessments on some projects, we have used it, and it is scalable. The online mobile application works well. If I would like to run it offline, it takes time to migrate the data that has been fed offline into the server.

Alex: That is another area where we can help you with setting permissions so that each person only needs to download their data. Thank you, Talib, for taking the time to share all of these systems that you have been able to build with the software, especially at the scale that UNICEF is working in Syria.