X

Elastic, a technology company that powers search solutions and protects against cyber threats, faced several challenges in its data practice. Takashi Ueki, Director of Enterprise Data & Analytics at Elastic, identified multiple sources of truth leading to disconnected reporting, a BI platform strategy that was misaligned with organizational needs, and varying definitions making it difficult to accurately and consistently report. Elastic's distributed and remote nature, along with its diverse team, made it crucial for any solution to be relevant and personalized to a spectrum of needs, expectations, and skill sets. The company's data governance strategy needed to drive transparency, accountability, and engagement, and be embedded seamlessly into the day-to-day experience of its distributed workforce.

GreenFlex, a European leader in environmental management services, energy efficiency, and environmental impact management, was facing a significant challenge in managing and governing the vast amount of data it was collecting. The company gathers energy consumption data from its customers and uses machine learning to identify consumption anomalies and devise energy strategies. However, as the volume of data grew exponentially, GreenFlex needed a simplified way to manage and govern this data. The company needed to make the data easily and securely accessible for data exploration, business intelligence, and machine learning use cases. Additionally, GreenFlex was maintaining three unconnected workspaces for development, staging, and production workloads, leading to complications in dealing with security and access controls over the tables in each workspace. This design also led to issues with data availability across these workspaces.

Grip, a company that serves e-commerce businesses shipping perishable goods across the U.S., processes hundreds of thousands of orders through its platform each month. The company's challenge was to process and interpret a variety of data points to make the most effective shipping recommendations. The data came from multiple sources including Shopify, ShipStation, different warehouse management systems, APIs for weather data, carrier pricing and delivery time tracking, and customer support systems like Zendesk and Dynamics. The company needed to consolidate this data to suggest the best carrier, ideal refrigerant and insulation, packaging and material, and other shipping logistics. Additionally, once a delivery was made, Grip needed to publish analytics so customers could see orders that went out, where they went, areas of the country that are bottlenecked, and areas of the country that are performing well.

The Hershey Company, a renowned name in the retail and consumer goods industry, was facing challenges in data management and analysis. Despite reaching a revenue milestone of $10 billion in 2022, the company was struggling with disconnected data sources, which prevented a single, consistent view of their data. This was a significant obstacle in making fast, data-driven decisions and staying ahead of market changes. The company aimed to build a Commercial Data Store (CDS) to serve as a single source of truth for commercial data across the entire organization. However, the complexity of Hershey’s data environment was further compounded by the fact that the company maintained separate data platforms to handle data for major retail customers.

Honeywell, a global provider of industry-specific solutions, is under increasing pressure to reduce energy use, lower costs, and improve efficiency. Their Energy and Environmental Solutions division uses IoT sensors and other technologies to help businesses manage energy demand, reduce energy consumption and carbon emissions, optimize indoor air quality, and improve occupant well-being. This requires Honeywell to collect vast amounts of data from millions of buildings worldwide. These buildings are equipped with thousands of sensors monitoring factors such as temperature, pressure, humidity, and air quality. In addition to this, data is also collected from external sources like weather and pollution data, and information about the buildings themselves. At peak times, Honeywell ingests between 200 to 1,000 events per second for any building, equating to billions of data points per day. Honeywell's existing data infrastructure was struggling to meet this demand, making it difficult for the data team to query and visualize the disparate data to provide customers with fast, high-quality information and analysis.

Incite, a company that provides the Rapid OEM Automotive Data (ROAD) platform to the world's largest automakers, was facing significant challenges in processing the massive amounts of unstructured data generated by test fleets. This data is crucial for building safer autonomous vehicles. However, frequent performance issues made it difficult for Incite to deliver fleet behavior metrics that clients could easily consume. Additionally, customers had to spend hours manually searching individual files for the data they needed, which significantly slowed down product development. The process of ingesting terabytes of data and making it visible in end-user dashboards took two to three weeks, which was a major bottleneck in the development process.

InMobi, a company specializing in targeted mobile advertising, was grappling with the challenges of managing a complex legacy infrastructure and a multicloud data warehouse. The company's data processing requirements had escalated to 20+ terabytes per hour, leading to skyrocketing costs and the creation of data silos that hindered collaboration and data sharing. The proprietary nature of their multicloud data warehouse also posed significant challenges. InMobi's existing system was overly complex, prone to outages, and extremely costly to scale. The company realized that their current system was slowing down their ability to innovate and was keeping their engineering resources tied up in maintenance tasks. InMobi sought a single system that could address multiple issues, consolidate their disjointed systems into a single platform, and free up their engineers to focus on higher-value tasks such as developing machine learning and large language models.

Integra Life Sciences, a global provider of medical technologies, faced a significant challenge when the COVID-19 pandemic disrupted the medical supply chain. The company needed a comprehensive view of global supply and demand to ensure the availability of its products for elective surgeries. However, its aging data warehouse limited its supply chain agility, causing delays in accessing critical data on usage patterns, stock levels, and quality issues. The legacy warehouse system, IBM DataStage, was time-consuming and inflexible, hindering the company's ability to respond to changing needs swiftly. The company needed a solution that would provide timely insights into inventory and demand, enabling it to deliver its products more efficiently.

Intelematics, an Australian real-time traffic information provider, was grappling with the challenge of synthesizing massive amounts of data from various traffic and mobile sources. The company's ability to provide insights from historical and real-time traffic trends relied on converting 12 billion rows of traffic data into actionable insights. However, the native systems previously used were tedious and laborious, hampering speed, efficiency, and collaboration across different teams. The company recognized the need to move away from its legacy on-premises infrastructure to keep up with increasing customer requests. The challenge was to process 12 billion road traffic data points every 30 seconds, sourced from millions of sensors and IoT devices on commercial and private vehicles. The native systems impeded the speed and efficiency of the data pipelines used to process bulk data in real time, such as traffic patterns, vehicle movements, and extensive monitoring data.

Iterable, a company that helps brands optimize and humanize their marketing, was facing challenges with its data infrastructure. The company needed to build personalized and automated customer experiences for its clients, which required harnessing diverse, complex data sets and facilitating rapid prototyping of machine learning models. However, the infrastructure they initially built with AWS native tools, including EMR, was resource-intensive, costly to maintain, and created significant operational overhead. This made it difficult for Iterable to scale the level of data ingestion and rapid prototyping of machine learning models needed to support its customer requirements and respond quickly to changes in the market. Furthermore, the company's AI solutions had to account for diverse data variables, drifts in the model, new regulatory changes, and a growing demand for more privacy protection.

Over the past decade, the broadcast television industry has undergone significant changes, largely due to the rise of streaming services. These changes have led to shifting viewer expectations, with people now expecting to be able to access a wide range of high-quality programming at any time and on any device. ITV, a British public broadcast television network, faced the challenge of meeting these changing expectations while also managing vast amounts of content data generated by nearly 40 million viewers. The company formerly relied on multiple legacy data platforms, which resulted in data fragmentation. Its data team was similarly fragmented across marketing, commercial advertising and product experience with its own technology stack. When the company launched its new digital strategy, it became clear it would need to modernize its platform and undergo a massive digital transformation with data at the core. ITV sought a platform that would allow it to consolidate its data sources and use analytics, machine learning, rule-based algorithms, and other tools to understand viewer expectations and behavior and improve the user experience.

Johnson & Johnson, a global consumer goods and pharmaceutical provider, faced significant challenges in managing its supply chain data. The company's growth through acquisitions led to a fragmented data system with disparate priorities and unique configurations. Data was largely being extracted and analyzed manually, limiting opportunities for speed and scalability. The disconnection was negatively impacting customer service and impeding strategic decision-making. The company also faced the challenge of optimizing inventory management and costs on a global scale, which required accurate and abundant data. The inability to understand and control spend and pricing could lead to limited identification of future strategic decisions and initiatives, potentially missing the opportunity to achieve $6MM in upside.

Kaltura, a company providing live, real-time and on-demand video SaaS solutions, faced the challenge of building a near real-time event pipeline. The data team was tasked with creating a new data product based on streaming events sent from users’ devices. This pipeline would need to capture events and write them directly into a data lake, detecting anomalies and notifying stakeholders of spikes in the number of events. The data engineering team, which had recently transitioned from supporting primarily the company’s cloud TV unit to serving the entire company, was also tasked with replacing the legacy infrastructure with a new data lake platform.

BizCover, Australia’s largest online business insurance provider, was facing a significant challenge in connecting data from various sources. The company's team of engineers had to build unique connectors using their own code, each requiring 40 to 80 hours of engineering time. This approach initially worked when connecting and syncing data from their database and Google Analytics. However, as the number of data sources increased, the task became overwhelming. BizCover needed to pull data from over 20 data sources into its centralized Snowflake data warehouse, with each source requiring its own connector. The company’s data engineers were managing this largely manual process, and BizCover needed to disseminate the insights they were gaining from the data across the core business more efficiently.

Blend, a fintech startup, was facing a significant challenge with its data ingestion process. Despite having adopted a modern data stack approach with Redshift at its core, getting data into and out of the data warehouse was proving to be a complex and time-consuming task. The process of pulling a single column from Salesforce or changing a field could take weeks, limiting access to time-critical data. The team was unable to prototype and rapidly iterate, and had to release straight to production to test their solutions, causing further complications for the operations team. As the company expanded, new tools like Asana, Marketo, and Lever were introduced to manage workflows and processes, each requiring data to be synced inside them to be effective. With the data engineering team’s limited bandwidth, they did not have the capacity to maintain a rapidly expanding list of SaaS platforms. This led to a decision point: commit to in-house tooling, or look for external providers.

CarOnSale, a disruptive pan-European platform for car dealers, identified data as a key differentiator in their market. As an online platform, they aimed to cut through the complexity of traditional car trading by harvesting and analyzing data around car auctions. This would provide them with unique market intelligence. The company recognized the need for a centralized architecture, hosted in the cloud, to collect and analyze data at speed and scale. They explored different options to support ELT (Extract, Load and Transform) as opposed to the traditional ETL approach. After selecting Snowflake as their cloud-based data warehouse, they needed to find the ideal data integration solution. Aynaz Bagherynezhad, Data Team Lead, had used Fivetran in a previous role and when Snowflake recommended it as the best way to connect to data sources, it confirmed her own experience.

Code2College, a nonprofit organization aimed at helping minority and low-income students achieve tech/STEM careers, was facing challenges in managing and analyzing student data. The organization's data, including student attendance, grades, and teacher input, was kept in spreadsheets or gathered by word of mouth. This approach was inefficient and time-consuming, especially when specific data on a student's performance or an overall view of the student population was required. The organization used Salesforce for operations and Canvas as a learning management tool. However, extracting information from these platforms to answer a single question would require a day's work, which was untenable given the small size of the data team. The team wanted to centralize their data using Google's BigQuery data warehouse tool to streamline retrieval and expedite responses to student needs. However, the challenge was how to transfer data from platforms like Salesforce and Canvas into BigQuery.

Condé Nast, a global media leader with 37 brands reaching millions of consumers, was faced with the challenge of managing and monetizing trillions of data points generated from its digital assets. The company lacked a central mechanism for managing and maintaining data integration sources, making data not readily available to consumers downstream. The demand to integrate more sources globally continued to grow, and pulling data into the data lake with custom scripts was cost prohibitive. Each marketing technology platform had its own API, data structure and other properties that required its own custom script. Creating the connectors on the fly and managing them on an ongoing basis wasn’t scalable, posing a significant challenge to the company.

Coupa, a Business Spend Management (BSM) company, provides a cloud platform that digitizes and consolidates spending information across various sectors, creating actionable insights into spending behavior. However, Coupa faced challenges with its own data about its platform and customer usage. The data was siloed, impeding better insights and decision-making. The process of collecting this data and making it accessible to the relevant personnel was complex, costly, and resource-heavy. Coupa had invested in a data team to manage its data, with the goal of pulling data from various sources into a single place for creating actionable insights. However, the analytics strategy was immature and largely consisted of ad hoc procedures. If a UX designer wanted to know how customers were interacting with a particular feature, they’d have to request the engineering team to build a script from scratch, a process that could take weeks.

Chris Klaczynski, a Marketing Analytics Manager at Databricks, was tasked with supporting the primary marketing objectives of driving pipeline generation, growing the database, and improving ROI. However, as Databricks rapidly expanded, the need for centralized and documented data became more and more apparent. Data silos were appearing around the business, including on Chris’ marketing team, where data was stored in its own data warehouse. It was critical that Chris’ newly-built dashboards were supplied with trustworthy, timely data for marketing operations to keep running smoothly. However, without dedicated engineering resources, and in the face of a rapidly expanding marketing team, scaling with demand became next to impossible. Databricks faced a number of challenges with their traditional data warehouse, including issues with their Salesforce and Marketo pipelines, issues appending data natively to existing tables, and schema changes that were always breaking pipelines, resulting in outages and stale, untrustworthy data.

Daydream, an early-stage startup, provides financial insights to stakeholders across modern businesses. The company's business modeling and planning tool democratizes access to financial information by bringing together processes and data sources that are typically siloed. However, the Head of Engineering for Daydream, Shubham Sinha, faced a critical decision. The success of the startup hinged on its ability to move massive amounts of data from its customers’ cloud-based business systems, each with their own login credentials and access challenges, into the Daydream platform for analysis. The two options were to either ask its customers for login credentials to their business systems, posing a potential security risk, and use custom-built data pipelines to onboard data or to rely on Fivetran to broker the credential sharing exchange and onboard data using its pre-built data pipelines. Maintenance also posed a challenge as each cloud platform has its own APIs, processes, and data structures, many of them requiring custom integrations through scripting.

The Denver Broncos, a successful pro football team, faced a significant challenge in maintaining their data pipelines. The team's Senior Director of Ticket Strategy and Analytics, Clark Wray, and his lean team were spending excessive time on home-brewed data integrations. Whenever an original data source or API changed, it would disrupt the data connections they had built, often halting the flow of information for hours. If these issues were not addressed promptly, the business risked operating on inaccurate data. Additionally, the team was constantly adding new data sources to communicate and reach the next generation of fans. It was crucial for them to connect and centralize their email data in Dynamics 365, marketing automation data in Eloqua, and fan feedback in Qualtrics.

DOUGLAS, a leading premium beauty platform in Europe, was facing a significant challenge in its journey to become a 'Digital First' business. The company's existing infrastructure and processes, particularly around Business Intelligence (BI) and data analytics, were not up to the mark. The systems for collecting data were scattered, and there was an overreliance on spreadsheets and manual input, which were not scalable. This lack of a centralized, automated data collection and analysis system was hindering the company's growth and its ability to gain valuable insights from its data.

DPD Polska, a leader in the Polish courier market, was facing challenges with its existing data management system. The company was using a series of on-premises PostgreSQL and Microsoft SQL Server databases to track its trucks, parcels, and people. However, the array of custom SQL databases was preventing DPD from producing timely reports, meeting disaster recovery time objectives, testing new data and analytic products, scaling up its revenues, and increasing its customer base. For instance, one of DPD’s databases had three different usage contexts. The company was in need of a log-based replication solution that would not impact its source systems. The main pain points were the replication time lags, the risk of errors in manual data distribution, and the need for more flexibility, greater reliability, and higher operational scalability.

Engel & Völkers, a prestigious broker of premium residential property and commercial real estate, was facing a significant challenge in integrating various data sources. The company's data engineering team was inundated with requests from different departments for data integration. The process of creating custom solutions to respond to these requests was resource-intensive, leading to prioritization of tasks and inability to cater to the growing number of requests. The company was in dire need of a tool that could reduce the effort required to integrate new data sources and enable faster data integration, thereby promoting wider adoption of self-service analytics within the organization.

Frontify, a platform that helps companies grow their brands, faced a significant challenge in building a single source of truth for their data. The company needed to understand how people interacted with their platform to optimize user experience and resource allocation. However, their data analytics team was small, and their data infrastructure was unstable. They relied on custom Python scripts to pull data from business applications into a MySQL database, which often resulted in slow, incomplete data. Their BI tool was user-unfriendly and slow, causing reluctance among employees to use it. The data team was burdened with the task of updating reports and dashboards. To address these issues and become truly data-driven, Frontify needed a scalable and powerful data stack that could be accessible to everyone.

GroupM, a global media agency based in Oslo, was facing challenges in collecting and analyzing data for their clients. The agency, which serves over 200 clients and provides shared services for other agencies in the group, was using Supermetrics to pull marketing data directly into Google Sheets. However, this method was proving to be inefficient and problematic. Pipelines would occasionally fail due to hard-to-detect issues, and there were formatting problems with the spreadsheets as well as manual errors. Preparing data for analysis in Google BigQuery, GroupM’s data warehouse, was labor-intensive, and clients were demanding faster access to more insights. One client, with a broad business portfolio spanning retail and hotels, was looking for dashboards that could handle historical data analysis as well as day-to-day reports. GroupM was determined to find a more robust solution.

Hashtag You, a brand builder in the direct-to-consumer e-commerce sector, faced a significant challenge in leveraging and structuring data within its organization. As a data-driven company, the use of analytics in marketing, product and customer analytics, and operational analytics was crucial to its business model. Initially, Hashtag You implemented several piecemeal solutions through Google Sheets with self-created data pipelines. However, the company soon realized the need for a centralized and scalable approach to data ingestion. The challenge was not only to establish robust data pipelines but also to connect new data sources quickly and easily. The company needed a solution that would allow non-data specialists to make these connections. Furthermore, the company had to manage advertising across various platforms, combine marketing and webshop data, link with other data pipelines, and analyze campaign performance.

HOMER, an early learning company, was facing significant challenges with its data management. Despite the company's data-driven roots, the data and analytics architecture was considered to be at a foundational stage when Joe Nowicki joined as Vice President of Data and Insights in February 2021. The company's data team was spending a significant amount of time building ETL pipelines, a laborious and time-consuming process. Flattening and maintaining Stripe data was costing the team dozens of hours each month, preventing them from adding value to the wider organization. The legacy data practices had led to a feeling of distrust, with leaders unable to depend on dashboard views and unclear business intelligence. Access to timely data was critical but lacking. There had been multiple visions and revisions of the entire data infrastructure, leading to the selection of Databricks’ Delta Lake, to set HOMER up for future Machine Learning applications.

Houseware, a software development company with less than 20 employees, was facing significant challenges in providing a platform and toolkit for its customers to build internal data products. The company's goal was to go beyond the scope of general analytics and data visualization tools, delivering metrics such as ARR, NRR, customer churn, conversion rate, and other KPIs. However, they were struggling with a lack of data insight, reliability, and availability. Their marketing campaigns were inefficient, and they were unable to turn data into customer retention optimizations. The trust in data was decreasing due to errors. Users had to learn data analytics tools and database methods, such as table joins, and develop custom metrics from scratch. Data dashboards and analytics often broke down, took too long to produce results, or required too much custom programming. Poor APIs and data pipelines limited the types of analytics that developers could construct to meet customer needs. The tools produced insights without any actionable recommendations, and building data connectors required lots of programming time and effort.