Measuring Impact in a Locally Led Network - Lessons from Building a Shared System
- HostFay Candiliari
- PanelistPedro Martínez
About the webinar
About the webinar
How do you build an impact measurement system for a diverse, locally led network without losing local ownership or creating unnecessary burden for the people expected to use it?
In this session, we are excited to welcome Pedro Martínez, Monitoring, Evaluation & Impact Lead, at WeRobotics who shares the Flying Labs Network’s journey from redesigning its Theory of Change and Indicators Framework to co-creating, testing, and implementing a shared impact measurement system using ActivityInfo.
Bringing together perspectives from both WeRobotics and Flying Labs, Peter reflects on what worked, what had to change, and what remains challenging.
The session focuses on the practical lessons emerging from the process:
- designing around the questions that matter,
- simplifying data collection,
- co-creating with local teams,
- supporting adoption across very different organizations,
- improving data quality,
- and moving from documenting activities toward stronger outcome and impact evidence.
View the presentation slides of the Webinar.
Agenda
- Starting with the measurement problem: moving from the Theory of Change and indicators to the information actually worth collecting.
- Building and testing together: how Flying Labs helped shape the system through co-creation, testing, and simplification.
- From implementation to adoption: what happened once the system moved into regular use, why adoption has been uneven, and what has helped.
- Moving from activities to impact: the limits of activity data and the development of a broader impact measurement ecosystem using self-evaluation, governance, and stakeholder feedback.
- Lessons from both sides: reflections from WeRobotics and Brazil Flying Labs on what they have learned and what they would do differently.
Is this webinar for me?
- How do you move from a Theory of Change and the Indicators Framework to creating a shared impact measurement system for a diverse, locally-led network, without losing local ownership?
- What are the lesson learnt from both perspectives?
Attend our webinar to listen to WeRobotics answering these and more questions!
About the presenters
About the presenters
Pedro Martínez is the Monitoring, Evaluation & Impact Lead at WeRobotics, where he leads the development of M&E and impact measurement approaches for WeRobotics and the global Flying Labs Network. His work focuses on building practical evidence systems for decentralized and locally led organizations, connecting data collection, analysis, learning, and impact measurement in ways that are useful to the people producing the information.
Pedro has more than ten years of experience working with data, M&E, and community-based programmes, including nearly seven years with UNHCR in Ecuador and Guatemala in information management, data management, and field roles. He holds a Master’s Degree in Human Rights, Democracy and Globalization from the Universitat Oberta de Catalunya and a Bachelor’s Degree in Social Education from the Universitat de València.
Transcript
Transcript
00:00:00 Introduction to WeRobotics and the Flying Labs Network
Pedro welcomes everyone to the session and notes that Diego, the coordinator of Brazil Flying Labs, was unable to attend, so Pedro will be presenting the journey alone. He begins by explaining the work of WeRobotics, a nonprofit organization dedicated to supporting local experts who use technology for social good. The organization operates on a localization model, rooted in the belief that the people best equipped to solve local issues are those actually in the field who deeply understand their own situations and ecosystems. WeRobotics supports these local organizations and links them with global partners, creating a virtuous circle of local experts leading initiatives with the backing of global stakeholders to build equitable collaboration and partnership ecosystems.
One of their flagship initiatives is the Flying Labs Network. This is a network of over 40 innovation and knowledge hubs located primarily in the Global South. These hubs work to strengthen local expertise in drones, data, and artificial intelligence, aiming to drive social impact through technological solutions, knowledge sharing, and the development of local ecosystems. Pedro also mentions that WeRobotics recently published their 10-year impact report detailing the achievements of the network over the past decade.
00:03:34 Redesigning the theory of change and indicator framework
Pedro provides context on how the impact measurement system was built. When he joined WeRobotics a year and a half ago, the organization was undergoing a significant learning and reflection process. They reviewed their strategy and Theory of Change, realizing that most of their initial goals had already been achieved. Consequently, they decided to pause and reflect on how to improve their operations. At the same time, the Flying Labs Network had grown exponentially. What started as a proof of concept in three countries had expanded to over 40 hubs handling more complex projects and engaging with a wider array of stakeholders.
This growth prompted a complete redesign of their monitoring and evaluation approach. They developed a new Theory of Change to outline the kind of world they wanted to see and their role in supporting those changes. Because their ecosystem is highly interrelated, their new Theory of Change took on a unique, cell-like structure rather than a traditional tree structure, placing their co-creation model at the center alongside several long- and mid-term outcomes. Following this, they built a detailed indicator framework to determine exactly how they would measure these desired changes.
00:08:02 Determining what information is worth collecting
With the Theory of Change and indicator framework established, the team faced the challenge of determining the minimum viable information worth collecting. They needed to figure out how to build a shared understanding of impact across a highly diverse, locally led network. Pedro explains that Flying Labs are not traditional country offices operating under a vertical power structure from a headquarters. Instead, they are independent organizations operating under a franchise model.
Anyone can apply to become a Flying Lab, and the current network includes universities, private businesses, and NGOs. Their common ground is a commitment to doing social good for their local communities. While this diversity is one of the network's greatest strengths, it presents significant challenges for monitoring and evaluation because the labs all do very different things, making it difficult to design a single system that works for everyone.
00:10:11 Assessing the data landscape and system requirements
The first step in building the new system was assessing their existing data and information landscape. The previous monitoring and evaluation system had served them well initially, but as the network expanded and the Theory of Change shifted, challenges arose. The old system was highly ad hoc, relying on individual "Master Sheet" spreadsheets for each Flying Lab to track projects and impact, alongside various documents stored in Google Drive. This setup made it incredibly difficult to see the big picture and understand how the network was performing as a whole.
They decided against simply digitizing their old spreadsheets, opting instead to rethink the entire system from scratch. They also wanted to solve the issue of repeatedly asking the labs for the same information, a common problem when working with localized organizations that have their own governance structures. They built a mock-up of a collection system, aiming for a flexible setup that could accommodate the diverse network without imposing rigid structures. They determined that the new system needed to be relational, utilize common classifications, incorporate relevance rules, and establish a monthly reporting routine.
00:15:18 Selecting the right platform
The team explored several options for hosting their new database. They quickly discarded the idea of building a custom system from scratch, as their small team of 13 people could not manage the ongoing development and maintenance burden. They also considered using form-based designs to mimic their old system in a more structured way, but feared this would replicate their previous issues.
Ultimately, they decided they needed an external platform and chose ActivityInfo after assessing several private companies against their criteria. ActivityInfo provided a relational database without requiring them to manage infrastructure, hosting, or maintenance. Crucially, it allowed for a tiered permissions system, enabling the Flying Labs to own their individual data while WeRobotics maintained a comprehensive view of the entire network. Familiarity with the platform, the team behind it, and the quality of their documentation also played a role in the decision.
00:17:57 Co-creating the system with users
Pedro emphasizes that the best decision they made was co-creating the system with the actual users. In his previous roles, information management teams typically developed systems in isolation and handed them over to users, which always caused issues. To avoid this, they assembled a representative group of Flying Labs from Kenya, Mozambique, Togo, Bangladesh, Brazil, and Costa Rica, ensuring a mix of regions and organizational types (universities, private companies, and NGOs).
Through four long co-creation sessions and a couple of test rounds, this group helped shape the database's design, the information to be collected, and the specific wording and categories used. The focus shifted from asking if information could be useful to asking if it was useful enough to justify collecting it. Pedro learned to curb the data professional's urge to collect as much as possible, adhering to the principle that data without a clear purpose should not be collected.
The Flying Labs completely transformed Pedro's initial mock-up, significantly simplifying the database and workflows. They reduced the number of required fields, recognizing that forcing users to log data they couldn't easily access would only hurt overall data quality. A major lesson was that more data is not better if its quality cannot be trusted. Furthermore, allowing users to play with the database revealed design flaws that seemed intuitive to the designer but were confusing in practice. Co-creating the system fostered a sense of ownership among the users. Pedro also notes that while ActivityInfo requires no coding skills, having someone on the team who understands basic database design is highly beneficial.
00:23:52 A tour of the Flying Labs log
Pedro provides a visual tour of the database, noting that WeRobotics' own impact management system is housed in the same place, which is incredibly useful for understanding how the network is doing and how WeRobotics interconnects with it. The main data collection system, called the Flying Labs log, features a lean, parent-child structure. Each Flying Lab has a single parent record containing its core information, connected to five interrelated subforms.
One subform collects basic information on team members, their roles, and their specific skills. This feature greatly improves network collaboration; if a lab needs help with a specific topic like drone hardware or machine learning, they can search the database to find and reach out to peers with that expertise. Another subform tracks partners, mapping out the ecosystem of local and global stakeholders. Because the database is relational, these member and partner subforms connect directly to the projects and training subforms, allowing the network to see exactly which individuals and organizations are collaborating on specific initiatives.
00:27:55 Testing, training, and rollout
After building the system, the team moved into testing, training, and rollout, which Pedro notes is where the real work begins. They conducted intensive training sessions across various time zones to accommodate the global network, training around 70 people and establishing two focal points per Flying Lab to ensure backup coverage. They built a sandbox mode where users could play around and make mistakes safely. A subsequent testing phase with the entire network provided valuable insights that led to further refinements.
The system was officially rolled out to production on April 1st. Pedro shares several recommendations for a successful rollout. First, provide clear, step-by-step, screenshot-based documentation rather than long, dense manuals, making it easy to direct users to specific solutions when they encounter issues. Second, utilize in-form guidance by adding brief explanations directly beneath complex questions within the platform, saving users from having to consult the external guide. Finally, provide continuous support and troubleshooting to help the network adopt the system.
00:31:14 Driving adoption through data visibility
Adoption proved to be the most challenging aspect of the process. After an initial surge of excitement and data logging upon deployment, engagement stalled due to summer holidays and inconsistent participation across the network. Instead of simply pleading with the labs to log their data, WeRobotics shifted their strategy to making the work visible and building a habit of regular logging.
They created a Power BI dashboard featuring an overview of the entire network's performance alongside detailed pages for individual Flying Labs, showcasing their projects and stakeholder engagements. This visualization was a game changer for adoption. Logging increased substantially as the Flying Labs began using the dashboard as a practical work tool to demonstrate their activities and impact to others.
00:33:38 Moving from activities to impact measurement
While the Flying Labs log successfully captures activity monitoring, measuring long-term impact indicators requires a broader approach. To address this, WeRobotics is implementing two additional tools to create a more holistic monitoring and evaluation ecosystem. The first is the Flying Labs Global model, an annual governance and self-evaluation tool that is being integrated directly into ActivityInfo. The second involves testing ActivityInfo's external survey feature to collect data and feedback from third-party stakeholders and beneficiaries.
To minimize the reporting burden on the Flying Labs, WeRobotics set up hidden calculated fields that automatically pull relevant information from the Flying Labs log directly into the self-evaluation tool. This integration ensures the labs can spend less time logging data and more time working in the field with their communities. Pedro concludes the presentation with a quote summarizing their philosophy: the goal is not to build a perfect database, but to create a system that allows a network to see, use, and learn from its own evidence.
00:36:26 Q&A session
The host thanks Pedro for the presentation and confirms that the recording and presentation slides will be made available on the website. A participant asks if Afghanistan is included in the countries covered by Flying Labs. Pedro clarifies that there is currently no Flying Lab in Afghanistan. However, the network grows organically, and any organization from any country can apply to join, provided they meet the network's requirements.
Ali asks what mechanisms ensure the digital system captures not only indicator progress but also the underlying reasons for success or failure, enabling adaptive program management. Pedro explains that alongside closed quantitative fields, they include several fields specifically designed to collect qualitative information on learnings, successes, and failures. This qualitative data, combined with the new third-party stakeholder surveys tied to specific projects, provides a comprehensive view of performance. The team reviews this mix of quantitative and qualitative data on a regular basis to make necessary adaptations to the program.
Vincent asks how they handle the specific information needs of individual Flying Labs if they require indicators separate from what is collectively collected. Pedro responds that because Flying Labs are independent organizations, they maintain their own internal monitoring systems for projects that don't perfectly fit the shared network system. However, WeRobotics is exploring the possibility of giving some Flying Labs direct access to ActivityInfo so they can build their own mini-databases for unique indicators, tying them into the broader ecosystem.
Vincent also asks for examples of gathering evidence rather than just tracking activities. Pedro reiterates that their approach relies heavily on collecting qualitative learnings alongside quantitative metrics. Furthermore, they conduct regular reflection sessions where Flying Labs meet with community coordinators to review the dashboard data, identify missing data points, and discuss what can be learned and adjusted. This continuous loop of data collection and reflection is what transforms activity tracking into actionable evidence. The host then wraps up the session, thanking Pedro and the attendees, and encourages participants to check the website for upcoming webinars.
Sign up for our newsletter
Sign up for our newsletter and get notified about new resources on M&E and other interesting articles and ActivityInfo news.