Data Engineering & Infrastructure
In modern data workflows, few tools have made a bigger impact than dbt and Snowflake. dbt brought software engineering practices into the world of data transformation, while Snowflake offered a powerful and flexible platform for storing and processing data at scale.
With dbt now directly integrated into the Snowflake interface, both tools work together more seamlessly than ever. This combination opens the door to simpler workflows, fewer context switches, and a faster way to build trusted data models right where your data lives.
In this post, we share our initial impressions of this new feature. We found that it simplifies development, enhances workflow efficiency, and provides a more unified experience for managing data transformations. Although the feature is still evolving, its current functionality offers practical benefits for teams using dbt Core with Snowflake, marking a significant step toward faster, more reliable, and collaborative data engineering.
What are dbt and Snowflake?
Before diving into the new integration, it is essential to understand the strengths of each tool on its own.
dbt (short for data build tool) has revolutionized how data teams manage data transformations by applying software engineering best practices such as modularity, git version control, and testing directly to data models. It enables analysts and engineers to write data models with SQL, which is maintainable, testable, and easy to collaborate on, thereby helping to build trustworthy data pipelines more quickly. dbt is an open-source framework that any team can host. Additionally, dbt Cloud is available as a product that enhances dbt’s capabilities at scale for thousands of companies.
On the other hand, Snowflake is a cloud-native data platform designed to handle large-scale data storage and analytics with speed and simplicity. Its unique architecture separates computing and storage, allowing teams to scale resources independently and pay only for what they use. Snowflake's intuitive SQL interface and rich ecosystem have made it one of the most popular choices for modern data warehousing.
Together, these tools address two key aspects of the data workflow: transformation logic with dbt and scalable, performant compute plus storage with Snowflake.
Using dbt Inside Snowflake: Features and Impressions
Snowflake now offers native support for dbt within its UI, specifically through workspaces, allowing developers to work directly with dbt models without leaving the current workspace. The integration is based on dbt Core, enabling teams to run commands while leveraging visual tools and deeper platform integration. Below is an overview of the current capabilities and experience using dbt in Snowflake.
Project Setup and git Integration
Although connecting to a git repository is not strictly required to use dbt within Snowflake, it is highly recommended for teams that want to collaborate effectively and maintain proper version control. Snowflake supports:
- git provider authentication using a personal access token.
- Using a dbt project versioned in a git repository.
- Version control through git branches, allowing teams to manage changes and collaborate on different development efforts.
This integration enables teams to track changes and collaborate in a structured way using standard git workflows. However, git commands are limited to Snowflake’s controls, which only allow two actions: pull and push all the changes into the remote repository. Any other special git command is not yet available through workspaces.
Dependency Management and External Packages
To install third-party dbt packages, like dbt-utils, users must first configure:
- A network rule to allow outbound connections to the package location.
- An external access integration in Snowflake to enable secure retrieval of packages.
Once this setup is complete, packages can be installed using the dbt deps command within the Snowflake workspace. This behavior mirrors the experience of using dbt Core and helps maintain consistency across development environments.
Model Development from the Workspace
The Snowflake workspace provides a focused environment for developing dbt models directly within the platform. The built-in editor supports dbt syntax and project structure, allowing users to write, modify, and organize their models efficiently. Features like automatic formatting, model file navigation, and quick querying contribute to a streamlined development experience.
Command Execution and Workflow Automation
The Snowflake workspace supports a variety of dbt commands directly from the UI, such as dbt run, dbt build, dbt test, and dbt deps. This enables users to execute workflows within the same environment where development occurs, enhancing efficiency by consolidating tasks into a single interface. It also allows for easier validation and iteration during model development.
There are certain commands, such as dbt debug, dbt docs, and dbt parse, that the UI does not currently support. However, these commands are not strictly necessary for basic usage and interaction with the dbt project.
Visualizing Model Dependencies
The DAG view stands out as a convenient addition, which displays the dependency graph of the dbt models and offers a great way to visualize data lineage. This visualization makes it easier to:
- Understand how models relate to each other to quickly track downstream and upstream dependencies.
- Assist in diagnosing inefficient data models to promote scalable design and enhance model performance.
Compiled SQL and Output Inspection
The platform includes an option to view compiled SQL for each model, allowing users to inspect the final queries after Jinja and macro processing. While this feature is not yet functional due to a known issue in the current release, it shows promise as a valuable debugging tool that can speed up the development process once completed.
Development Experience and First Impressions
Working with dbt inside Snowflake provides a smooth and integrated experience for developing transformation logic. By centralizing model editing, execution, and visualization, teams can work faster and reduce the overhead of switching between tools.
The setup process does require some configuration upfront, particularly regarding external access and git integration. However, once in place, the workflow feels familiar to dbt users and introduces several efficiencies.
While the feature will continue to improve, its current state offers a solid foundation for modern data development within the Snowflake platform.
Benefits and Limitations of the Integration
Benefits
The native integration of dbt within Snowflake offers several clear advantages:
- Streamlined workflows: By working entirely within Snowflake UI, data teams avoid switching between multiple tools, improving productivity.
- Improved collaboration: git integration supports version control and team coordination, which is essential for maintaining robust data projects.
- Faster development cycles: Real-time feedback and direct access to compiled SQL help accelerate model debugging and iteration.
- Secure dependency management: Configuring network rules and external access integrations ensures reliable package installation and maintains security standards.
Limitations
The integration is still maturing, however, and has some room for improvement:
- Limited git functionality: git integration in Snowflake provides only basic version control features, enabling essential management directly within the platform. However, more advanced actions, such as resolving conflicts and reviewing detailed commit histories, must be performed outside of Snowflake. This limitation means teams engaged in complex workflows or frequent merges will likely need to rely on external tools to manage their git repositories fully.
- Less intuitive CI/CD automation: Automated testing and deployments rely on integrating the Snowflake CLI with external CI/CD tools, such as GitHub Actions. This setup requires additional manual configuration and may add complexity for some teams.
- Less polished user experience: While the integration covers core dbt functionality, it lacks some user-friendly features found in dbt Cloud, such as interactive documentation, detailed job history, and streamlined testing interfaces. This means users might currently encounter a steeper learning curve and less convenience than dbt Cloud offers when managing projects.
Conclusions
Snowflake’s native support for dbt provides a practical and efficient solution for managing data transformations within a unified platform. Though the functionality continues to mature, its current state already delivers tangible improvements in workflow efficiency and simplicity. This integration is a valuable advancement for organizations relying on dbt Core and Snowflake, paving the way for more streamlined and agile data engineering operations.
We encourage data teams to explore this new feature and experience firsthand how it can enhance their development workflows. By integrating dbt directly into Snowflake, teams can unlock greater productivity and simplify their analytics processes without leaving their data platform.
About Aimpoint Digital
Aimpoint Digital is a market-leading analytics firm at the forefront of solving the most complex business and economic challenges through data and analytical technology. From integrating self-service analytics to implementing AI at scale and modernizing data infrastructure environments, Aimpoint Digital operates across transformative domains to improve the performance of organizations.
Author
