dbt Labs released a slew of exciting new dbt Cloud features at Coalesce 2023, making the question of dbt Core vs dbt Cloud even more pressing. Here’s what you need to know about the enhancements and how to decide whether dbt Core or dbt Cloud is right for you.
Sunrise or sunset? Beach or mountains? Sun or snow? Crunchy peanut butter or smooth? These are age-old questions that most of us have encountered from time to time. And for those of us in analytics engineering and data engineering, we have a new one to add to the list…dbt Core or dbt Cloud?
The question of dbt Core vs dbt Cloud is becoming more and more common—and with the recent announcements at Coalesce 2023, it’s more important than ever. dbt Labs unveiled some major new dbt Cloud features to solve some of the biggest problems organizations face. As dbt significantly builds up and enhances the capabilities of its cloud offering, businesses need to take the time to determine which offering is right for them.
In this post, we’ll dive into when and why dbt Cloud might be a better fit. First, let’s understand the basics of dbt Core and dbt Cloud.
dbt Core
dbt Core, the open source version of the tool, has long been the go-to solution for transforming data in your warehouse. It empowers analysts and engineers to develop locally on their machines, build modular SQL and python queries, apply transformations, and document their work effectively.
With the ability to run dbt models directly in your data warehouse, dbt Core seamlessly integrates into existing data workflows. The software itself is free, but it shouldn’t be understated that there are associated costs. Organizations looking to run dbt Core in production are required to manage the deployment infrastructure themselves, which means both time and money spent on hiring people to maintain the infrastructure.
Price tag: Free*
*There are costs associated with the people and technical time required to maintain dbt core implementations.
dbt Cloud
dbt Cloud enhances the dbt experience with features tailored for the cloud, such as the ability to schedule and orchestrate dbt runs, manage dependencies, and promote models between different environments.
dbt Cloud extends the capabilities of dbt Core by providing a cloud-based platform for analytics engineering. This fully managed service offers a collaborative environment and convenient UI, enabling teams to work seamlessly on their analytics projects.
Price tag: See chart below.
| Product | Software License Cost | Seats | Model Builds |
| dbt Cloud Developer | Free | 1 developer | 3K/month |
| dbt Cloud Team | $100/seat | Up to 8 developers | 15K/month |
| dbt Cloud Enterprise | Custom | Custom | Custom |
When Is dbt Cloud a Better Fit?
When dbt Cloud first launched, its biggest advantage over dbt Core was its built-in orchestration capabilities and its ease of use for newer developers. However, since then, dbt Labs has made great strides in building up the capabilities of dbt Cloud. Now, more and more businesses are turning to dbt Cloud for a more governed, accessible, and scalable approach to data transformations.
Next we’ll take a look at some common pain points and scenarios where dbt Cloud would be a better fit vs dbt Core. To start, we’ll cover a few of the exciting new scenarios made possible because of the slew of features released at Coalesce 2023.
Scenario 1: Data Literacy and Team Scaling
- Pain point: Onboarding is a chore. It’s hard to get new users up to speed because documentation is hard to find, non-existent, or inconsistent. User provisioning is complex and your engineers need to support each new user with installs and configurations for passwords and environments. And when new users are unfamiliar with Git, your engineers are slowed down by maintaining a custom CI/CD process to check PRs.
- Solution: Git integration, Slim CI, and dbt Explorer (Team and Enterprise only)
- Available on: dbt Cloud Developer, Team, Enterprise
dbt Cloud eliminates the need for local installs and configurations because they’re all stored and configured in Cloud. The platform also consolidates role-based user access, environments, continuous integration, and job deployment into one application and UI. dbt Cloud also has built-in git integration and comes with dbt’s Slim CI for simpler and faster continuous integration.
The successor to dbt Docs is dbt Explorer, dbt Cloud’s new knowledge base and lineage visualization experience.
dbt Explorer makes it easier than ever to share context, troubleshoot issues, and reuse assets across different parts of the organization. It’s built for the type of scale seen at larger organizations. And for organizations that need a unified documentation solution but would otherwise choose the Developer plan, dbt Docs is available for Cloud providing smaller organizations with a cost effective solution to improve collaboration, transparency, and efficiency.
Scenario 2: Large Projects and Teams
- Pain point: Your organization suffers from bottlenecks and slow development due to a large, complicated monolithic project spread across many different teams and business areas.
- Solution: dbt Mesh and cross project references (Enterprise only)
- Available on: dbt Cloud Developer, Team, Enterprise
dbt Mesh is a pattern (rather than a product) made up of several dbt features, including:
- Cross project refs (Enterprise only)
- dbt Explorer
- Model Governance – contracts, versioning, and access
- dbt Cloud CLI
Until now, there hadn’t been a way to effectively manage the dependencies, governance, and workflows between multiple dbt projects, which often led to project silos or single, giant, monolithic, and complicated projects.
With dbt Mesh, projects can be separated into their logical parts and seamlessly built on each other. Interfaces between them are governed by contracts, versions, and access controls that are baked directly into dbt. This enables teams to control the data pipelines that are directly relevant to their work while enabling a central data team to set global standards for governance and maintain visibility on end-to-end dbt lineage. With the Enterprise level, product users can also easily use the built-in ref function to reference models in other projects.
Scenario 3: Highly Technical Personnel Resistant to a New IDE
- Pain point: You have a highly technical team that is resistant to changing its workflows. They are already comfortable and efficient with their own IDE and don’t want to get locked into the Cloud IDE
- Solution: dbt Cloud CLI
- Available on: dbt Cloud Developer, Team, Enterprise
The new dbt Cloud CLI allows developers to use any command line (CLI) or terminal of their choosing (VS Code, Sublime Text, etc.) fully backed by dbt Cloud. Developers get the familiarity of their CLI of choice while still reaping the benefits of dbt Cloud and eliminating the hassle of manual configuration and authentication like you might experience with dbt Core. Just as you can on the IDE, the CLI allows for deferring to production for unbuilt dev models upstream, saving users a lot of time.
Scenario 4: Consistent Metrics via the Semantic Layer
- Pain point: End users are constantly complaining about metric inconsistency. Because business logic and metric definitions are locked up in a BI tool or defined differently across multiple BI tools, end users are spending a lot of time writing code to create multiple slices and dices of different tables
- Solution: dbt Cloud Semantic Layer
- Available on: dbt Cloud Team and Enterprise
The Semantic Layer aims to bring business logic and metric definitions into one single source of truth that lives inside dbt Cloud right along with your code. Since acquiring Transform, dbt has revamped their semantic layer (which is now powered by MetricFlow) to enable more complex metric definition and querying to be done efficiently at scale.
Some new features include:
- Expanded data platform support and performance optimizations for BigQuery, Databricks, Redshift, and Snowflake
- More complex metric types out of the box
- New aggregations
- More flexible metric definitions
Long-time Benefits of dbt Cloud
As you can see, Coalesce 2023 was a game-changer for organizations evaluating dbt Cloud. Now, here are three long-time benefits of dbt Cloud vs dbt Core.
1: Built-in Orchestration
- Pain point: Your team doesn’t have the time, resources, and/or technical prowess to integrate an orchestration tool into the mix.
- Solution: Job deployment and job scheduler
- Available on: dbt Cloud Developer, Team, Enterprise
Rather than building and maintaining your own infrastructure, users can create, run, and schedule jobs right in the integrated development environment (IDE), ensuring the timeliness and reliability of data transformations. The scheduler can handle Cron-based execution and event-based executions triggered by things like pull requests or APIs.
You can use the job scheduler to handle various tasks such as to:
- Queue jobs
- Create temporary environments to run the dbt commands required for those jobs
- Provide logs for debugging and remediation
- Store dbt artifacts for direct consumption/ingestion by the Discovery API
2: Compute Costs & Project Visibility
- Pain point: Compute costs keep increasing, but investigating and pinpointing the root cause is difficult and time consuming.
- Solution: dbt Cloud Discovery API
- Available on: dbt Cloud Team and Enterprise
Every time dbt Cloud runs a project, it generates and stores information about the project such as details about your project’s models, sources, and other nodes, along with their execution results. With the dbt Cloud Discovery, API developers can query this information to get a better understanding of the Directed Acyclic Graph (DAG) and the data it produces.
The API can be accessed with ad-hoc queries, custom apps, partner ecosystem integrations, and by using dbt Cloud features like model timing and dashboard status tiles. It can be used for a variety of purposes like:
- Investigating model performance or data quality
- Discovery to understand assets
- Governance to see who developed and who uses certain models
- Development to better understand downstream impacts
3: Outbound Webhooks
- Pain point: Your downstream workloads have a complex setup or extra steps such as dropping log files in S3 and ingesting them. You have to create and manage a polling workflow to get simple yet important information about your dbt job statuses.
- Solution: Outbound Webhooks
- Available on: dbt Cloud Team and Enterprise
Outbound webhooks allow dbt to send notifications about your dbt jobs to your other systems, allowing you to get the latest information on your dbt jobs in real time. Your other systems can listen for these events to further automate your workflows or to help trigger automation flows you have set up. Webhooks can be created using the browser based IDE or the dbt Cloud API.
So…Is dbt Cloud or dbt Core a Better Fit for You?
As you can see, dbt Cloud offers a lot of amazing features to tackle very common development problems for users of all skill levels and organizations of every size. The announcements made at Coalesce 2023 really show dbt Labs’ commitment to the product, and their focus on certain enterprise-level issues has me really excited. I can’t wait to see how organizations at the highest scale with hundreds of developers, thousands of end users, and tens of thousands of models use solutions like dbt Mesh and dbt Explorer to solve problems of complexity.
Here’s an at-a-glance comparison to help you evaluate the solution that’s right for you based on the features you care about most.
| Feature | dbt Core | dbt Cloud Developer | dbt Cloud Team | dbt Cloud Enterprise |
| Model governance | ✅ | ✅ | ✅ | ✅ |
| Git integration | ✅ | ✅ | ✅ | |
| Slim CI | ✅ | ✅ | ✅ | |
| Cloud CLI | ✅ | ✅ | ✅ | |
| Job deployment and scheduler | ✅ | ✅ | ✅ | |
| dbt Explorer | ✅ | ✅ | ||
| Semantic Layer | ✅ | ✅ | ||
| Outbound webhooks | ✅ | ✅ | ||
| Discovery API | ✅ | ✅ | ||
| Cross project references | ✅ |
With enhanced features, seamless orchestration, and the promise of continuous innovation, dbt Cloud stands out as a great choice for teams seeking not just a transformation tool but a dynamic and evolving data analytics ecosystem.
Looking for a Partner with dbt Expertise?
Need help with your dbt strategy, implementation, or dbt Core vs dbt Cloud evaluation? As a dbt Labs Premier Consulting Partner, Data Clymer is here to help. Contact us today for a free dbt strategy assessment or check out our growing library of free dbt video tutorials.
