Databricks' Transition from Data Silos to a Unified Data Lakehouse

Chris Klaczynski, a Marketing Analytics Manager at Databricks, was tasked with supporting the primary marketing objectives of driving pipeline generation, growing the database, and improving ROI. However, as Databricks rapidly expanded, the need for centralized and documented data became more and more apparent. Data silos were appearing around the business, including on Chris’ marketing team, where data was stored in its own data warehouse. It was critical that Chris’ newly-built dashboards were supplied with trustworthy, timely data for marketing operations to keep running smoothly. However, without dedicated engineering resources, and in the face of a rapidly expanding marketing team, scaling with demand became next to impossible. Databricks faced a number of challenges with their traditional data warehouse, including issues with their Salesforce and Marketo pipelines, issues appending data natively to existing tables, and schema changes that were always breaking pipelines, resulting in outages and stale, untrustworthy data.

Databricks is a business that prides itself on helping data teams solve the world’s biggest challenges. It was founded in 2013 by the original creators of Apache Spark, Delta Lake and MLflow. Built on a modern Lakehouse architecture in the cloud, Databricks combines the best of data warehouses and data lakes to offer an open and unified platform for data and AI. The company has rapidly expanded, leading to the need for centralized and documented data. However, data silos were appearing around the business, including on the marketing team, where data was stored in its own data warehouse.