In a previous article, we discussed the existing challenge of bringing data into your data warehouse from a source that does not have an out-of-the-box connector from a data syncing service like Fivetran. In that article we presented one possible solution utilizing Data Clymer’s custom data connector framework. Since this framework is highly customized to our client’s exact needs, it requires extensive fine tuning; or, development hours.

In this article, we’ll be discussing a different approach, Fivetran Cloud Functions. Specifically:

  1. Their serverless design.
  2. Benefits to this approach.
  3. A client use-case for Simplecast – a podcast hosting & distribution platform.

Like all innovative organizations, there are specific platforms used to create an edge and competitive advantage. The key is your ability to take actions based on the insights you glean from these services. That’s where things can get tricky and overwhelming. Which vendor natively connects to this source? How much ongoing maintenance is needed? How long will it take to get this data into my data warehouse?

Well, Fivetran’s Cloud Functions are here help answer such questions. This feature allows users to leverage popular serverless cloud computing products such as AWS Lambda, Azure Functions, and Google Functions to bring in data from a source that isn’t currently supported by one of their 180 native connectors.

The approach works in 3 simple steps:

  1. Write a small function to fetch data from your custom source.
  2. Host a function on a serverless platform.
  3. Connect Fivetran to capture data from the platform and load it into your data warehouse.

Under the covers, Fivetran is taking care of scheduling of the invocation for the cloud function as well data inserts, updates, and deletes. Resulting in quicker development times, accelerating time-to-live, and eliminating server maintenance. Another notable benefit to this approach is it allows you to develop hyper reusable code. This can be beneficial to organizations who run a multi-cloud architecture, migrating cloud providers, or want to share the scripts with others. Meaning, because you’re hosting the cloud function that is running the underlying code, you’re able to easily shift to another cloud function in the case of changing service providers. 

In Action: Simplecast

A client recently approached us needing to track the performance of their editorial strategy. Their desire was to slice data and get metrics around engagement across several types of media, including podcasts. Our client was using Simplecast, a podcast hosting and distribution service, as one method of interacting with their audience. Fortunately, Simplecast has fantastic backend analytics that they expose to their clients via an API.

Our client was already using several of Fivetran’s out-of-the-box connectors to bring in other data sources to their BigQuery data warehouse. After reviewing the Simplecast API, our team concluded Fivetran Cloud Functions was the best fit, because it allowed us to quickly integrate a new data source into their data warehouse. Hitting 12 endpoints for this single API, we had the desired data available after authorship of a single python function that only interacts with the Simplecast API in just 3 days. Even better, we can now utilize this code for future clients also using Simplecast because of it’s cloud-agnostic extensibility. 

If your organization is wanting to ingest Simplecast data into your data warehouse, our team is already a step ahead!