00:00:00 Introduction and agenda

Thanks, Fay. Thank you so much, and let's get started because we have a lot of material to cover. We are going to look at measuring impact or outcomes. Both of those types of indicators often use surveys to measure results, and I want to give you some best practices for setting up the process from beginning to end in ActivityInfo, based on some of the work that we've done with our users at a large scale.

I am going to break it down into about four different stages. First, we will look at the design. How do you think about these surveys not just as a one-off exercise, but as an element in your information management system or monitoring system? Then, we will turn briefly to sampling and look at how to do some very basic sampling as needed. Third, we will look at actually doing the fieldwork and collecting the data using ActivityInfo's mobile app. Finally, we will turn to ActivityInfo notebooks for doing some analysis of the data that we've collected to draw some conclusions about the results within our participants.

00:01:51 Designing the impact evaluation

Let's start with design. Where does quantitative impact evaluation fit in? This is a topic on its own, and we did a webinar last year on measuring quantitative impact which you can find on our website. As you will recall if you were in that session, what we are looking to do is to evaluate an indicator with the program that we're implementing versus without the program. We want to know what would have happened to our participants if we hadn't implemented our program.

Essentially, we are asking for a counterfactual. We don't have a multiverse where we can let time pass with and without the intervention, so we have to find good ways to draw comparisons. In some cases, you can look at your participants before and after, but you really need to consider alternative evaluations. Today, we will be looking basically at before and after surveys.

To do that, we are going to use a case study from our development assistance database. It is an example drawn from Pennsylvania, and we will be looking at how to improve maple syrup yields for farmers. For this program, the key indicator that we are looking to measure is that the total yield per farm increases 20% among participants. The second is that the yield per tap increases. When you are harvesting maple syrup, you drill taps into trees to get the syrup in the spring. The more sap you can get from each of those taps, the more effective and efficient your farm is going to be.

00:05:38 Database structure and indicators

I want to talk about these impact surveys not just as a one-off exercise, but in the context of a full information system that we are following along over the course of several years. We want to look at not just surveys that measure outcomes, but also collecting information in our trackers on different activities, training events, and outputs. We want to link both the surveys and the trackers back to the registration information.

When we register participants in our program, we have a lot of valuable information about them. When it comes to doing the surveys, we don't want to have to collect all that information again. We want to be able to include that data in our analysis to see how participation in different activities influenced outcomes. You can answer these questions when you use a relational database like ActivityInfo to connect all of these pieces together.

Let's open up ActivityInfo and take a look at this database. This comes from our template library, so you can get started with this template called "Development Assistance Project." I have folders for these different pieces of information about my program, starting with the registry where I am tracking farms and individuals. I can view that on a map to see where the farms are located. We also have a list of trackers where we can track all the trainings that we've done.

If we want to know the outcome or impact of these activities, we might not be able to go back and collect data from every single farm due to resources. The reason to sample is that programs often lack the money or time to talk to everyone, so it can be very useful to take a small survey of the farms annually or quarterly.

00:10:46 Understanding sampling concepts

We did a series of webinars on sampling in 2021, but we will do a quick refresher. Two key terms to know are population and sample. The population is the whole group that we are talking about, whereas the sample is the specific households or farms that we are talking about. In our maple syrup example, our population is the farming households or total farms in the area. In my dummy dataset, I have about 55 different farms that we are working with.

The sample is a smaller group because we perhaps cannot afford to visit all 55 farms. Before we can do the sample, we need a sampling frame. A sampling frame is a list of all of the members of the population. In a scenario where you are sampling only from among your participants, your registry form is simply your sample frame. In a second scenario, where you are looking at participants and non-participants to compare, you would need a separate form that includes all participants as well as a list of comparable non-participants. Today, we will look at scenario one, sampling only from among our participants.

Once we have our sample frame, we can randomly select the individual farms that we are going to include in our survey. In ActivityInfo, we are going to create a chain from our registry to a sample form, and then link a questionnaire to that sample. This way, our survey questionnaire can always be linked back to the sample to get weights for inclusion probabilities and to ensure we are only interviewing people we selected. We can also pull in data from our registry into the questionnaire and analysis.

00:16:45 Drawing a random sample with Excel

How do you draw a sample? How do you go from a sample frame of 55 farms and choose 10? You need a tool for this. You can do this with Excel, R, or SPSS. Excel is a good tool for simple random sampling or basic stratified sampling. If you want to do cluster sampling, you are probably going to need R or SPSS. Today, we will focus on Excel.

We are also going to calculate the weights of our sample items. The inclusion probability is the chance that each person in your population has of being selected. If we have 50 people and sample 10, everyone has a 20% chance (0.2 probability) of being selected. Weights are the reciprocal of your inclusion probability. If everyone has a 20% chance of being selected, that means everyone has a weight of five. Intuitively, every respondent represents five people in your population.

Let's go back to ActivityInfo. I have my list of farms, which is my population and sampling frame. I am going to export this to Excel. To randomly select 10 farms, I will add a new column called "Random Number" and use the function =RAND(). This gives me a random number for every member of the population. Then, I will sort the list using the random number as the sort key. The top 10 farms will be my sample. This is a simple random sample where every farm has the same chance of being selected.

00:23:35 Setting up the sample form in ActivityInfo

Now that I have my sample list in Excel, I am going to create a form in ActivityInfo to hold my sample. If you are doing the same survey in several waves (e.g., Baseline, Year 1, Year 2), you can keep this sample all together.

First, I will add a serial number generated by ActivityInfo to identify each sample. Then, I will add a reference field to the "Farm" registry so I can link back and pull in information. Third, I will add a field for the "Wave" to indicate if this is the Baseline, Year 1, or Year 2 survey.

I will import the 10 farms I selected in Excel into this new form. The name of the farm will be automatically matched to my registry, forming the link. I also need to include the inclusion probability. Since I have 55 farms and selected 10, every farm has an 18% chance of being selected. I will import this value (0.18) as well. It is better to import this rather than use a calculated field because different waves might have different sample sizes or inclusion probabilities.

00:28:50 Creating the survey questionnaire

Now we can create our survey questionnaire. This will be linked back to our sample. The surveyors will work off the sample list we give them, entering the sample ID. I will make the Sample ID field a key field to ensure interviewers don't interview the same farm twice.

I will add a calculated field to confirm the name of the farm. This field will follow the path from the Sample ID to the Farm reference to the Name of the farm. This acts like a VLOOKUP or a left join, showing the interviewer the name so they can be sure they are interviewing the right person. Then, I will add my survey questions, such as "How many taps did you drill this year?"

00:31:26 Planning the fieldwork

Now let's look at the actual fieldwork. Our sample form provides the roadmap for our interviewers. Because it is linked to our registry form, we can pull in information to help plan. In the table view, I can select columns to bring in related information about the farm, such as the mailing address, city, zip code, and the owner's name and phone number.

This creates a roadmap for fieldwork. I can sort by city or zip code to plan the route. You can export this and print it out for your teams, giving them the Sample ID number they need to plug into their survey and the contact details to find the farm.

00:35:24 Managing users and mobile data collection

To invite surveyors, go to Database Settings and then "Users." You should think about the role you want to give. I have a role here for "Field Staff" which allows them to view, add, and edit records, but not delete them. If you are working with a large number of interviewers, you might want to restrict permissions so they can only see their own data.

When inviting someone, pay attention to the language. If you have translations of your surveys, ensure you choose the correct language for the user so their invite and mobile app interface appear in that language.

Regarding the mobile app, surveyors can install ActivityInfo from the Play Store. They log in using their email. If they need to work offline, they can download the database to their device. They can then open the survey form, enter the Sample ID (which will pull up the farm name for verification), and fill out the questionnaire. Once they have an internet connection, the data will synchronize automatically.

00:44:27 Analyzing results with notebooks

For the last step, let's look at analysis. We want to use the weights we set up to extrapolate our sample data to the population. In our survey form, I will add a calculated field to calculate the weights, which is 1 / inclusion probability.

I will add a Notebook to put together different types of analysis. First, I can add a pivot table to see my sample size (count of interviews) broken down by wave. Then, to see how many farmers are cultivating maple syrup in the total population, I will not use the count. Instead, I will drag the "Weight" field into the measures and sum it. The sum of the weights will equal the total population size.

For example, if I look at the question "Did you set taps?" and use the sum of weights, I can estimate that 44 farmers in the population are cultivating maple syrup. I can then create a bar chart comparing Year 1 and Year 2 using the weights. This allows us to see the change over time across the entire population, not just the sample.

00:51:43 Q&A session

Q: How do you do random sampling digitally? A: You can use Excel as we demonstrated, or tools like R and SPSS.

Q: Does sorting by ascending or descending matter in Excel sampling? A: No, because we are using random numbers, the sort order doesn't matter. Taking the top 10 or bottom 10 will still result in a random selection.

Q: What is the cost of the software? A: Pricing is available on our website. It depends on the number of users. For example, up to 75 users is roughly 4,800 Euros per year.

Q: How do you draw a sample frame for complex data analysis using R? A: You can export your data via the API directly into R. From there, you can use the base R sample function or the survey package, which is very powerful for cluster and stratified sampling.

Q: How do photos work? A: You can add an attachment field to your form. On the mobile app, this will allow you to take a photo with your camera directly from the field and store it in the database.

Q: How do you handle sampling for widespread and scattered populations? A: This relates to complex sampling. I recommend watching our "Complex Samples" webinar, which covers strategies for difficult-to-reach populations.

Thank you so much for joining us today.