00:00:04 Introduction and roadmap planning

Okay, so before we get started, I'm going to give the floor to Jamie to go through a lot of the features in detail, but I did want to just take a moment to talk about how we plan a roadmap, what we're going to share today, and what you can do to influence or provide information on this roadmap. First question: how do we pick what we're going to develop this year and next year? We have three main inputs. The first is our mission: what we're trying to accomplish as an organization. The second big one is customer feedback; we're listening to users all the time. The third is customer-funded engineers. When customers provide us with resources to hire additional engineers, we give them the ability to prioritize what those engineers work on. We have limited resources, and about two-thirds of your subscription fees for ActivityInfo go into research and development and improving the software, but we do our best to prioritize based on these three areas.

ActivityInfo and BeDataDriven’s mission is to provide the social sector with a standard, easy-to-use, comprehensive data management platform so that as many organizations as possible can become data-driven, ultimately to achieve better outcomes for people worldwide. We believe that by having an integrated platform, we make it easier to actually use that data to make better decisions to guide your programs. We try to test each of the ideas and possibilities against this mission.

We also listen a lot. Our idea board on Jira now stands at 271 ideas. We track who is requesting these features and how many customers among our customer base are requesting them, and we do our best to prioritize those that we see will have the broadest impact across all of those using ActivityInfo. We also have bilateral meetings with you and sometimes bring multiple customers together to talk about a feature to help inform the design.

The development process for any of these features starts with a specification, a detailed document laying out what we're trying to achieve. We move into user experience design, which helps us think through not just how it looks, but how people interact with it to make it as easy and obvious as possible. From there, the coding and testing start, followed by translation and documentation before it finally goes live. We have a continuous release process for our Software as a Service, so every Tuesday there's a new release.

00:05:00 The data lifecycle

I have tried to organize all of the things that we're currently working on according to the data lifecycle. This process goes all the way from collection to processing, storage, management, analysis, visualization, and finally interpretation, which is the user's role to make sense of this data. We are going to go through these stages and look at the features that are currently in development or at least planned on our roadmap. We are going to focus on the ones that we have designs for. Just because we don't mention something in this presentation doesn't mean that it's not on our roadmap; every quarter we reprioritize and adjust.

00:06:49 Data collection features

We have put the data collection features into two groups. One is field types, such as multi-select reference, geographic area, and Matrix data entry. The second is a bigger group of features around drafts, conflict resolution, offline mode, and offline collection links.

The first feature is multi-select reference fields. Currently, reference fields are limited to a single selection. We are going to extend this to allow you to have multiple selections of reference fields included as a separate field type. You will see a search bar in the data entry interface that will allow you to look up records you would like to reference in another form. You can have multiple selections shown as a card list, and we will allow you to reorder and remove them as necessary. We will also extend the mobile interface to support multi-select reference fields.

Next is geographic points. We are going to have a suite of improvements to better capture the exact geographic point you would like, whether using automated features for entering the current location or manually drawing a point on a map. We will also support the manual entry of latitude and longitude coordinates. Beyond points, we are going to extend our functionality to include geographic lines and areas. You will be able to use multiple recordings of your current location to draw a specific line or area on a map, manually draw them, or use an automated recording of your location at a certain interval.

00:11:02 Matrix data entry

We have a new field type called Matrix. This matrix data entry will allow you to enter quantitative data and measures in a tabular format in a simple, easy-to-use interface. You will be able to define a matrix in a similar fashion to how you define sub-forms, by defining your dimensions of measures that you want to collect for specific data sets. For example, you can add a row dimension for seasons and a column dimension for sports to capture how many people are performing these sports per season.

On data entry, you will see a simple, tabular interface included inline within the data entry form. This allows you to enter numbers quickly without having to go to a separate form or enter multiple entries for each dimension. The idea is to treat these almost like a subform, so you have a list of values rather than expanding the matrix to 30 or 40 fields, which would make it difficult to analyze. You could then use SUM and other functions to aggregate those in the main form or use those column dimensions in the pivot table.

00:14:24 Drafts and conflict resolution

We have a big set of features coming up related to drafting. Drafts help solve the long-form problem where you might not be able to complete a form with 200 questions in one sitting. We also want to use this to change the way offline mode works. Currently, when you complete data entry offline, we use optimistic conflict resolution where the last change to the server wins. As ActivityInfo grows, we are seeing more cases where people are working with data about the same entities, making conflict resolution important.

In the new design, you will see a drafts table at the top of the view. We compute validation for each draft record on the fly, indicating whether it is invalid, valid, or has a warning. If a record is invalid, we annotate the individual cells with an error message. We also support referencing other draft records. You will be able to do a batch submission of all valid drafts. If corrections need to be made, you can start a submission wizard to go through each individual record and make them valid before submitting.

Regarding conflict resolution, if you are modifying a draft locally and someone changes the value on the server, we are pushing to resolve this locally. You will be able to merge your updates from the server or take your own version of the draft as the accepted version.

00:23:00 Offline collection links

We are introducing offline collection links, which use the mechanisms we have been implementing with drafts to support the ability to collect multiple anonymous records via a collection link interface offline. This allows a user to take a collection link, store data locally, and submit multiple records once they return online without requiring a user account. This is a solution for casual labor hired to do surveys.

You will be able to have multiple collection links open, name them, and share a QR code. You can also see stats on when the link was opened and how many records have been added. Users can download the collection link as a Progressive Web App (PWA), saving the website to their local mobile device for easy access. On the mobile interface, collectors will see a drafts table view of the records they have added offline and can resolve any invalid records before submitting to the database.

00:27:52 Storage and processing improvements

We have done a lot of work on storage and processing over the last two years. We invested heavily in our own optimizing query engine that performs calculations and data retrieval uniformly in the web browser, on the mobile phone, and on the server. In December of last year, we completed a project to change the way our servers process data to handle a larger volume, pushing the limit from 200,000 records per form to one million records.

The next project is processing data in the browser and on the mobile device. We use a standard called IndexedDB to store data, and we are looking to improve performance for very large datasets on the device, perhaps by finding ways to synchronize less data.

00:31:17 Data management: Duplicate scanning

We have a group of features for bulk editing and cleaning data. The first is duplicate scanning. We have a new section in the Table View menu called Tools with a "Scan for duplicates" option. You can select and scan certain fields on a form for potential duplicate values. You can set matching sensitivity to exact matches or probabilistic output.

Once the scan is complete, you can review the results grouped into potential sets. You can ignore records that are not duplicates, remove records that are mistakes, or select records to merge. When merging, you choose a primary record and then resolve the differences between the values in the duplicate group. This allows you to merge records and update all references to those duplicates in other forms. We are also looking at expanding this to handle synonyms and translations in the future.

00:41:22 Inline editing and spreadsheet mode

We have requests for the ability to edit a value quickly without opening the full data entry interface. With inline editing, you can double-click a field in the table view and update the value in a detail pane. If the change induces validation errors, you will be prompted to resolve them in the data entry interface.

We are also introducing a spreadsheet editing mode. This provides a spreadsheet version of all values that you can edit directly. You can add new rows, sort, filter, and delete rows. This mode acts as an input to the Importer; all validation, locks, and permission checks are respected when you apply the changes.

00:47:43 Form views

We are introducing a "Views" feature. A view is a configuration of your columns, filtering, sorting, and column sizing that you can save and persist. You can create personal views or shared views. You can also associate a view with a different visualization type, such as a map. You will be able to associate certain views with specific roles, so they show up by default for data collectors or administrators.

This gives us a structure to build out different types of views. One new view type is "Reader mode," which is a long-form view of records that allows you to scroll through and inspect information, including attachments and subforms, more densely. We also have an "Attachments view," which shows all the images or files from your database in one place for easy navigation.

00:55:10 Analysis and visualization

On the analysis side, we are working on dynamic filters on dashboards, similar to Power BI slicers. You will be able to define dimensions that apply to any reports on the dashboard, allowing you to filter by multi-selections or other data types.

We are also working on Tagging. This addresses the challenge of balancing localization with central control and analysis. As organizations roll out ActivityInfo to many countries, it becomes harder to aggregate data across different localized databases. Tagging allows you to add tags to fields, forms, folders, or database resources. You can then use these tags in reports to aggregate dimensional data across all your forms without needing precise naming conventions.

00:59:33 Future roadmap items

We have several items at earlier stages of design. We are looking to expand the formula language for things like geographic analysis. Qualitative analysis is another area where we are collecting use cases. For visualization, we want to add more customization options for branding and more chart types. We are also looking at thematic map templates.

Regarding languages, we are working on auto-translating qualitative data and supporting a "long tail" of languages to make it easier to work in local languages even if the system translation isn't available. Finally, we are exploring opportunities in artificial intelligence, particularly around auto-summarization, coding qualitative data, and cleaning data using large language models.

01:06:13 Q&A

Question: Regarding the Matrix field type, is the idea to bring in three components, for example, like SADD (Sex, Age, Disability) data? Answer: Yes, you can have more than two dimensions, but you can only have one column dimension. For SADD data, you could put sex and age in the rows and disability in the columns. You can have multiple row dimensions, but only one column dimension.

Question: When you apply a view to a role, does that effectively work as sharing the view? Answer: Yes, that is the way you would share a view. If you are an administrator with permission, you would share that with specific roles in your database. You can also make it a default view.

Question: Can you do a calculated field off this Matrix field type, like a sum? Answer: During data entry, it displays as a matrix, but for analysis purposes, it is treated as a subform. You would use aggregate functions like SUMX to summarize the data.

Question: Are there plans to extend auto-translation to the data we are receiving? It would be wonderful to not have to export and re-upload data to translate it. Answer: We just added Translated Fields, which creates a metadata link between fields. We want to implement the ability to view translated data on the fly in the details view. We could imagine a tool that auto-translates a field into another language and populates that extra field, which can then be updated and treated like any other data.