At AgileData.io we have a toolkit of patterns for data collection, we are mega stoked to add Dataddo as a core part of that toolkit

Dataddo brings 300+ data connectors which our AgileData Network partners can use to collect data from their customers’ Systems of Capture.

When we first started #AgileData, we decided we would focus on the messy and complex middle part of the data value stream.

We didn’t want to deliver the last mile, there were many great products that did that already and the table stakes to compete were high.

We didn’t want to build the capabilities to collect data from customers’ systems, from experience we knew how many data collectors we would need to build and maintain, plus we had an assumption that customers would already have technical teams that collected the data into a data lake for them. We assumed we would just read the data from that.

This second assumption quickly proved wrong. Organisations we talked to were too small to have a data team, that’s why they were talking to us, as a result they did not have technical teams doing the data collection process for them. They wanted us to solve their data problem, and this included the collection of their data.

As a result we have been building out our data collection patterns using a toolkit approach.

We started off by architecting four core data collections patterns we would support, Push, Pull, Drop and Stream.

As we got new customers who needed us to deliver a solution using one of those patterns we developed it.

We enabled customers to securely push data to us. We enabled customers to drop/upload csv, json and excel files. We enabled streaming of data to us.

When we needed to pull data from the customers’ SaaS systems, for example systems such as Shopify & Hubspot we experimented building these using a Meltano/Singer Tap pattern.

This worked well but it distracted us from working on patterns in the complex and messy middle.

We decided to find a partner that could offer this capability and integrate easily with our AgileData Patterns.

Dataddo is our partner of choice.

Our data collection toolkit now includes:

• Filedrop/upload of csv, json, excel files.
• Google Cloud Data Transfer Service, for Amazon S3, Azure Blob Storage, Youtube, Google Ads, ServiceNow etc.
• Dataddo, 300+ SaaS connectors, for Hubspot, Shopify, Xero, Stripe etc.
• Streaming, for Google Analytics to BigQuery, pushing of events to Google Pub/Sub etc
• Custom data collectors built by AgileData Network partners.

This last part of our toolkit is powered by AgileData Network partners who develop connectors for Systems of Capture where no options exist in the current toolkit.

For example one of our partners has just completed a data collector for Brightcove. This collector will be available to all our AgileData Network partners.

These data collection patterns provide a toolkit that allows our partners to reduce the complexity of collecting data, in a simply magical way!