00:00:04 Introduction

Thank you, Fay, for the great introduction as always. This is the third of our three-webinar series that we started implementing a couple of months ago. What we aim to do with the help of CARE, Missio Invest, and IREX is to get more in-depth into practical examples from organizations that implement surveys in their work in the field.

That is why we are here today, to hear how three different organizations with great and different profiles design and implement their surveys. We will understand their processes and have a nice conversation at the end of the session. Without further ado, I will pass it over to Haitham from CARE to explain what they are doing in Yemen in terms of survey implementation.

00:01:39 CARE Yemen: Survey design in a consortium

Hello everyone. This is Haitham, the Consortium MEL Advisor from CARE Yemen. Before we start our presentation, let me give you a brief introduction about the consortium project that we are currently implementing in partnership with ActivityInfo. Currently, at CARE Yemen, we are implementing the BRIGHTLY Consortium project funded by FCDO. We have a consortium of five partners: four international partners like CARE, NRC, Oxfam, Save the Children, and one national organization, YFCA. We are managing this project in around 1,000 sites and more than 22 districts. In this project, we are implementing key activities under food security and livelihood, building household resilience, community empowerment, and systems transformation. We focus mainly on unconditional cash transfer, conditional cash transfer, financial inclusion initiatives, and farming and livelihood development activities.

Given the fact that we have a couple of partners from different organizations and we are implementing a large project with a large amount of data to be reported, we contracted with ActivityInfo to have a centralized data management system. This allows us to combine information across all areas and enables the consortium partners to compare and contrast their achievements and data timely when needed. The plan was to have an online database for all partners to report data and activities.

We chose ActivityInfo because we needed real-time information to be shared across all partners and reported on time. We also have the ability to use data outside of ActivityInfo, for example, using phone surveys or paper-based interviews. We can use the open collection link to capture data online or offline, using a multi-language interface for the field team. We can do data management and processing timely, importing previous data directly into the system. Having all the data in ActivityInfo allows us to analyze and display real-time data in pivot tables and charts.

In terms of survey design and implementation, at the beginning of each survey, we do close coordination with all partners to conduct a comprehensive survey in all targeted areas based on the logical framework and M&E plan. We identify the outcome indicators that need to be measured and define the survey objectives. Once we define the objectives, we design the survey method and questionnaire in cooperation with other partners. We select the most appropriate method for data collection, determine the target population, and select the sampling approach. Before data collection preparation, we design the questionnaire to have a logical flow and use clear language.

We use harmonized instruments and tools for all partners to implement the planned survey. We pilot the survey before the activity is implemented in the field to reduce errors and data inaccuracy. We ensure security permits are available for the field team and implement quality control measures to address any issues identified before implementation. At this stage, we use the ActivityInfo data collection link. Once data collection starts, we ensure ethical considerations are met, obtaining informed consent and adhering to 'do no harm' principles. Finally, we clean and validate the survey data, analyze the results, and share findings with decision-makers for future programming.

There were some challenges reported during this process, namely security and access challenges due to ongoing conflict and instability in parts of Yemen. Obtaining survey permits on time is difficult. The sampling frame and selection of beneficiaries in hard-to-reach areas is a concern, as is potential bias. Ensuring the confidentiality and privacy of respondents in challenging environments remains a priority. Lastly, coordination with a large number of stakeholders affects effective collaboration and communication.

00:12:59 Missio Invest: Shopping for information

Hi everyone, I am very happy to be here today to speak about how Missio Invest uses ActivityInfo for surveys. Missio Invest is a social impact fund that has worked in Africa since 2016. We have disbursed loans of 23 million dollars across 96 loans in 10 countries. We work with institutions of the Roman Catholic Church to help them become financially sustainable. The borrowers we work with are very diverse, ranging from a group of sisters in Uganda to a highly sophisticated hospital in Nigeria. When we design surveys, we have to take into account who is answering them, as there is a very different level of sophistication among our borrowers.

We are very proud of the ripple effect we have. In 2023, we had 64 borrowers operating 2,881 social institutions that impacted 16 million people. While we might give a loan to an agricultural business, that farm will feed a school and provide income for a hospital. On average, our borrowers run over 40 institutions, and a loan to one has a ripple effect on the whole network.

Regarding survey design, I use the analogy of "Shopping for Information." Surveying is like sending someone grocery shopping. You need to make sure you buy the right things to cook a healthy dinner. You need a shopping list, which in business terms corresponds to a Theory of Change. For Missio Invest, our "healthy meal" includes resilience, food security, strengthening institutions, and shared prosperity. We also have specific sectors we work in: agriculture, healthcare, education, and microfinance. The questions we ask originate from the crossing over of our Theory of Change and the specific sector.

For example, in agriculture, we ask about farm productivity. In healthcare, we ask for medical information. We also have shared indicators across all sectors, such as employee data and economic impact. When designing a survey, keep in mind who is filling it out. Use easy language, no complicated abbreviations, and keep it short and simple. Cover the basics first so that if someone drops off, you still have the essential information. Group your questions by topic using subsection headers in ActivityInfo. Use relevance rules, formulas, and validation rules to ensure data quality, just as you would want high-quality ingredients for a meal.

For survey implementation, we share PDF handouts in advance to our borrowers because they cannot save a survey in ActivityInfo and return to it later. These handouts help them prepare the complex indicators. We provide significant support, including filling out the baseline in person with a country manager. The baseline survey is the trial run for the annual impact surveys. We track submissions as they come in and review the data in context, comparing it to baseline and planned data. Communication is key throughout this process.

Our top three takeaways are: First, design simply in terms of language and the number of indicators. Second, prepare your borrower so they understand what you are doing and why. Third, ensure data quality by looking at information in context and documenting the process.

00:28:04 IREX Youth Excel: Designing for implementation research

I represent the Youth Excel program at IREX. IREX is a global development and education organization working in more than 100 countries. Youth Excel is a USAID-funded, five-year, cross-sectoral program that empowers young people and youth organizations to use implementation research to strengthen development solutions. We have reached 3,000 people trained on implementation research and 110 youth-led organizations across 20 countries.

Our MEL system uses ActivityInfo to store and collect data. We extract this data, transform it, and load it into Power BI for analysis and reporting. We conduct three main types of surveys: reporting surveys, annual surveys, and targeted surveys such as training or post-event surveys.

When designing surveys, we focus on setting SMART objectives: Specific, Measurable, Attainable, Relevant, and Time-bound. Clear objectives provide a framework for data interpretation, clarity for respondents, and improved data quality. Good objectives also help avoid bias and determine how to treat sensitive questions. We collaborate as a team, including field teams and implementing partners, rather than making it a one-man show. We also use standardized questions to allow for benchmarking and comparisons.

For implementation, we consider resources such as staffing and materials. We ensure we have enough research assistants and backup materials like hard copies. Training is crucial; we train enumerators on the survey objectives, data collection protocols, data quality, security, and the platform itself. Logistics planning is also essential to ensure data collection happens within the scheduled timeframe. We also have a dissemination plan to decide how we will share the insights, whether through reports, briefs, or videos.

One challenge we experienced was low response rates when sending surveys directly to participants. We solved this by coordinating with local partners to follow up, which improved response rates. Another challenge is working in contexts with multiple languages like French, Kurdish, and Spanish. We translate surveys and then translate responses back to English. We also learned that piloting is very important to ensure formulas and rules in the online platform work correctly. Finally, using a user-friendly platform is key to success when resources for training are limited.

00:42:00 IREX Asas: Surveying in the education sector

I am the MEL Director for the Early Grade Education Activity, Asas, in Jordan. "Asas" means foundation in Arabic. Our goal is to advance national efforts to strengthen pre- and in-service education and improve kindergarten through Grade 3 foundational skills. We work with the public sector and aim to equip around 23,000 teachers and reach 1 million students. Our approach is a journey concerning teacher preparation, from when they are students to becoming teachers.

In addition to what my colleagues have mentioned regarding survey design, I want to highlight field access authorization. It is vital to ensure we have approval from the authorized entity to empower our team to collect data. We also must ensure participant consent. Participants need to know why we are doing the survey, the purpose, the benefits, and that their participation is voluntary and confidential.

When dealing with a heterogeneous group, we need to unify our tools. The flow, content, and language must be simple. For example, we recently conducted a Teacher Readiness Assessment. We identified five different target groups: graduates, university teachers, novice teachers, senior teachers, and supervisors. Identifying these groups allowed us to design the survey and choose the right approach. We decided on a mixed method, combining quantitative and qualitative methods.

For the qualitative part, we conducted in-depth interviews, collaborating with the Ministry of Education to access schools. For the novice teachers, we used a quantitative approach via ActivityInfo. We trained supervisors to disseminate the self-administered survey. This minimized bias because no one judged their input, allowing them to speak freely about the system. ActivityInfo was crucial for this quantitative approach, allowing us to download data periodically and conduct data cleaning simultaneously.

We also use ActivityInfo for event attendance and to collect critical data from university students. Collecting anonymous data from students is essential so they feel free to express their challenges. We are continuously working with ActivityInfo to improve our processes and data quality assurance.