00:00:00 Introduction

Hello and welcome. Today's webinar is the first of a series that will go through how we can design and implement a survey. We will cover the logistics of the survey on the ground, and in the final session in June, we will see specific examples from customers using ActivityInfo to design and implement surveys using ICT4D technology.

00:00:55 To survey or not to survey?

When we get into the process of designing a survey, the first component is to weigh the benefits of a survey versus the risks and the resources we need to dedicate. The basis of the whole implementation is the project proposal documents, specifically the theory of change, results framework, and the monitoring and evaluation (MEAL) plan. These act as cornerstones of survey implementation and dictate the objectives.

We implement surveys because we want to learn and improve our project implementation during the present phase or for future projects. We also implement surveys to evaluate to what extent we have achieved our higher objectives and to be accountable towards the beneficiaries we serve, donors, partners, and staff. The purpose is to collect accurate, non-biased, and relevant information to promote evidence-based decision-making.

There are four key principles to keep in mind. First, implement user-centered design by considering where our users are and how to consult the community. This includes sampling methods, such as whether to interview specific individuals or households. Second, include "need to know" information rather than "nice to know" information, guided by the MEAL plan and regulations. Third, collect reliable information through proper questionnaire formulation and trained data collectors. Finally, adhere to ethical standards, always keeping the "do no harm" approach in mind.

A survey should have a clear objective. If the objective is not identified, look into the initial project documents. We must evaluate whether a survey is the most appropriate way to gather information or if we can obtain it from other resources. Surveys require resources and specific sampling, so we must weigh the resources needed against the positive impact the survey will bring. We also need to consider timing; for example, if we provide cash assistance today, we cannot survey the impact tomorrow. Enough time must pass for the outcomes to manifest.

00:10:15 Questionnaire development

Questionnaire development refers to the actual questions, answers, formulation, structure, and ordering. The starting point is the project documentation: the theory of change, results framework, and MEAL plan. The MEAL plan includes indicators and measurement methods, while the data analysis plan provides more detail on cross-tabulation, comparison groups, and sampling techniques.

Surveys primarily contain quantitative questions—closed-ended or numerical questions. It is advisable to state the units, such as kilometers or hectares, to clarify the expected answer. Recently, mixed methods approaches have gained ground, combining quantitative information with qualitative insights. For example, following a Likert scale question about satisfaction with an open-ended "why" question can help identify components for improvement.

Field testing is crucial and should not be skipped, even with limited time. It allows for localization, ensuring the survey works in the specific context and indicating the feasibility of the questions.

00:15:42 Steps in questionnaire development

We can identify a step-by-step approach to development. Step zero is always the identification of objectives. Step one is aligning survey questions to these objectives. Step two involves reviewing the feasibility of the questionnaire, such as its length and the sample size required. Step three is reviewing the structure, format, and translation. Step four is the field test or pilot test, followed by incorporating feedback. Finally, we proceed to training enumerators.

00:18:00 Writing the content

When writing content, we rely on the theory of change. For example, in a cash-based project, activities lead to beneficiaries receiving cash, spending it, and ultimately meeting basic needs or reducing negative coping strategies. Survey modules often correspond to these theory of change or results framework headings. Following this breakdown helps ensure no components are missed.

It is advisable to consult with other teams internally, such as safeguarding or feedback and response mechanism teams, as they may need to collect specific information not included in the standard MEAL plan. Regarding length, try not to exceed one hour for adults and keep it under 45 minutes for children, or even less depending on their age.

For specific indicators, such as the reduction of coping strategies, resources like Indikit.net provide detailed guidance on recommended questions. This helps confirm question types and validation logic. For instance, if asking how many days in the past week a strategy was used, the answer must be between zero and seven.

00:25:02 Best practices for writing questions

Questions need to be precise and have one clear objective. Avoid "double-barreled" questions, such as "Was the doctor professional and helpful?" because the doctor could be professional but not helpful. Use unbiased, neutral language. Instead of asking "Why don't your children go to school?", formulate it as "What are the main reasons the school-aged children in this household are not able to attend school regularly?"

Avoid questions that result in heated answers or are culturally inappropriate. For sensitive topics, ensure the respondent is only asked about themselves or their dependents. Facilitate recall for complex questions or those referring to past incidents by reminding the respondent of the specific event before asking the question.

Refer to previous questionnaires or international guidance from USAID, UNHCR, OCHA, or sector-specific clusters. When writing multiple-choice options, ensure they are mutually exclusive and cover all potential answers. Always include "do not wish to answer" or "do not know" options to distinguish between a skipped question and a refusal to answer. For quantitative questions, use codes like 999 or 888 for non-responses.

00:30:17 Common problems and solutions

Consider a question like: "Which percentage of your average monthly income did you spend on hygiene products?" This presents several issues: it requires difficult mental calculation, it is ambiguous, and income can be sensitive. A better approach is to ask two separate questions: "What was your income during the past month?" and "What was the amount you spent on hygiene products?" This allows you to calculate the percentage later.

Another example is "How far do you live from the cash distribution site?" This lacks specificity regarding the unit of measurement (time vs. distance) and which site. A proposed solution is "How much time does it take you to reach the nearest distribution site?" with answers in minutes or hours. Similarly, avoid abbreviations in questions, such as "CBA," and instead spell out "Cash-Based Assistance."

00:34:58 How to format the questionnaire?

A well-formatted questionnaire minimizes errors and improves data accuracy. Group questions into modules to make administration easier. Order the modules and questions logically, starting and finishing with non-sensitive questions. This allows the enumerator to build rapport before asking difficult questions. Start with a unique identifier and a clear introduction covering who you are, the purpose, time required, rights of the respondent, and data protection measures.

Include instructions for enumerators, such as whether to read responses aloud or if a question is single or multiple select. When designing in Word, use a table format that includes the question code, the question itself, response options, skip logic, and validation logic. Ensure consistent formatting and leave space for observations.

00:38:30 Tips to ensure data quality

Two main components for data quality are skip logic and constraints. Skip logic ensures relevance; for example, only ask "Which feedback mechanism do you know?" if the respondent first answers "Yes" to being aware of a mechanism. Constraints restrict answers to expected values, such as ensuring a response about days in a week does not exceed seven.

00:39:30 Ethical considerations in design phase

We must adhere to the "do no harm" principle. The benefits of the survey must outweigh the risks. Ensure respondent privacy and confidentiality, and obtain voluntary informed consent. Respondents should know what data is being collected, stored, and used.

Be careful with the placement and phrasing of sensitive questions. For example, instead of directly asking "Do you gamble?", which can be insulting or putting the person in the spotlight, rephrase it to "How many times have you gambled in the past month?" and define what constitutes gambling. Make sensitive questions optional or include a "prefer not to answer" option to avoid missing data or forced responses.

00:42:55 Questions and answers

Is it mandatory to include a comparison group during baselines? The short answer is no. While a comparison group is ideal for impact evaluation, it depends on your research questions and resources. In humanitarian settings, it is often difficult to identify a comparison group if everyone receives help simultaneously. If you lack funds or access, you may not need a complex evaluation design; simple project monitoring might suffice.

What are the best survey tools for measuring knowledge of a group before or after a conference? For measuring knowledge, pre- and post-tests are standard. You can administer a test before the capacity building or conference and one after. If resources are limited, you might use a post-test with recall questions asking what they would have done differently before the training.

Can we record the name of the respondent? Yes, provided you have consent and a secure place to store the information. You must ensure data protection and not leave papers with names lying around. Often, if the respondent is already participating in a project, you may already have their details and do not need to ask again.

How do you handle sensitive questions to avoid low response rates? Clear communication at the beginning about why you are collecting data helps. Also, monitor how enumerators ask these questions; they may need training to ask them more comfortably. If a question is too sensitive and yields no answers, consider dropping it. Rephrasing to be less direct or making the question optional can also help maintain trust and reduce drop-off rates.