X

Schüttflix, a German logistics start-up, aimed to disrupt the traditional construction supply chain by digitizing the industry, which was primarily reliant on pen and paper processes. The company's mission was to enable data-driven decision-making by providing stakeholders with accurate and timely data. The challenge was to build a modern data stack that could tap into key data sources efficiently and reliably. Alexander Rupp, Head of Data and Business Intelligence at Schüttflix, was tasked with evaluating connectors that could meet these requirements.

Lendi, an Australian mortgage broker with over $12 billion AUD in home loan settlements, was facing a significant challenge in its data management. The company's business model, which involves offering customers a choice of over 2,000 loan products from more than 40 lenders, relies heavily on delivering the right experience to the right person at the right time on the right digital platform. This requires accurate insights into borrowers' needs and preferences, which in turn requires tapping into behavioral data on third-party engagement platforms such as Facebook, Google, and Bing. However, the data from these platforms were siloed and did not integrate easily. Even when the data could be brought into the same repository, the data structure was often inconsistent, necessitating data cleaning before it could be used. The responsibility of ensuring that stakeholders across the company could analyze and create actionable insights from this third-party data fell to Lendi’s Data Architect, Daniel Deng.

Paytronix, a customer engagement platform for restaurants and small businesses, was facing a significant challenge in managing and deriving insights from its data. The company was dealing with data from multiple sources, running on various databases, and in disparate formats. The data ingestion tool they were using was unreliable and missed many transactions, leading to a lack of trust in the underlying data. Additionally, the company was using a mix of Scala and PySpark jobs for data transformation, which was custom code and handwritten. This toolset was unable to keep up with the growing demands of the business, and a lot of time was spent on maintenance and break-fix support. The company wanted to focus more on experimentation, but the existing system was not conducive to quick proof-of-concept testing and rapid iteration.

Billie.io, a Berlin-based fintech startup, is revolutionizing the way businesses handle payments by providing instant financing for invoices and outsourcing the collections process and default risk coverage. However, the company faced a significant challenge in managing its data architecture. The company needed a solution that could handle the Extract, Load, and Transformation (ELT) process of their production database to the data warehouse efficiently and cost-effectively. The company also needed to avoid latency problems or Service Level Agreement (SLA) issues and prevent transformations from occurring too early. Furthermore, the company wanted to have fine-grain control over when things happen and awareness on tasks that comprise pipelines, their dependencies, and their execution.

Canva, an online design and publishing tool, was under pressure to grow its customer base across three service levels—free, pro, and enterprise. The sales, marketing, and engagement teams needed to identify targets, understand their behavior, and deliver the right message at the right time on the right platform. This required a 360-degree view of the customer across Canva’s digital properties and third-party platforms such as Google, Facebook, and other social media and SaaS tools. The main challenge was the lack of comparison insight. There was no way to analyze Facebook data against Google data or any other platform. Building a point-to-point architecture to pull data into competing platforms would be messy and expensive to maintain. The custom-built solution functioned as intended, but the demand for connectors to more data sources was growing, and the time and resources required to build these connectors were not scalable.

Parachute Home, a U.S.-based direct-to-consumer brand selling home essentials, was struggling to manage data from its two core systems, Shopify and NetSuite’s cloud ERP software. Shopify powered Parachute’s ecommerce platform and transactional process, while NetSuite triggered the fulfillment process. However, these systems were running in a siloed manner, making the data from both sides increasingly hard to manage. Parachute was using custom-built data loaders to connect into Shopify and NetSuite, but the results were inconsistent. There were data quality problems that resyncing rarely solved, and the absence of logs made it hard to identify issues. The time-consuming data ingestion process was hindering the brand's digital ambitions.

Paul Hewitt, a jewellery and accessory brand, was facing challenges in managing and analyzing its advertising spend due to the limitations of its existing tool, Supermetrics. The analytics team, consisting of three employees, had to manually enter data into spreadsheets to determine the most effective marketing channels, a process that was both time-consuming and prone to errors. To meet the needs of an increasingly complex supply chain, the company had invested in an ERP system, Microsoft Dynamics NAV, and began to make data available for analysis with Microsoft Power BI. However, the company wanted to take its data strategy to the next level by integrating data from across the business into one place on a cloud data platform, with the objective of transforming into a data-driven business.

Pitney Bowes, a global technology company that simplifies e-commerce, shipping, and mailing, was facing significant challenges with its data management. The company lacked high-quality, real-time data necessary for critical business decisions. Its Enterprise Information Management (EIM) team was grappling with siloed data, lack of scalability, and inefficient tech spending. Employees were resorting to pasting data into Excel spreadsheets for executive reporting and analytics, which often exacerbated the issues. The company was also experiencing downstream problems, such as late-arriving packages that impacted Service Level Agreement (SLA) targets. They lacked the sophistication to detect delays and notify customers in time, causing reputational risk. The COVID pandemic magnified these data challenges when online shopping increased tenfold, leading to a tenfold increase in parcel volume. The company's legacy data infrastructure was unable to handle event- and email-based data operations for 800 million packages per day. The data captured was critical, but aggregating and consolidating it to the central analytics warehouse took days, making it outdated by the time it reached the leadership team.

PopSockets, a retail and consumer goods company, was facing significant challenges with its data management and reporting processes. The company was struggling with reporting efficiencies and communicating insights across various departments, including Ecommerce, Marketing, Business Intelligence, Finance, Accounting, Supply Chain, and Operations. The lack of strict timelines for data refreshment and the tedious process of manually aggregating reports were hampering the company's growth. As PopSockets began to experience tremendous year-over-year growth and adopted an ERP system, the volume of data grew exponentially. The company was grappling with data silos, unscalable manual efforts to aggregate and store data in a single source of truth, and a lack of visibility into marketing data to understand the ROI of ad spend on various channels. PopSockets needed a scalable solution that would allow its small team of data engineers to build automated data pipelines for faster analytics and reporting.

PostNL, a mail, parcel, and e-commerce service provider in the Netherlands, United Kingdom, Germany, and Italy, decided to migrate its IT operations to the cloud to maintain its competitive edge and reduce costs. The company aimed to decommission its on-premises data centers and move its applications, infrastructure, and IT management to the cloud. The applications ran on Oracle and SQL Server, and where possible, PostNL wanted to replace existing bespoke software with Software-as-a-Service (SaaS). If a suitable SaaS replacement was not available, the company planned to implement the legacy and bespoke software on top of cloud-based infrastructure and platform services (IAAS and PAAS). PostNL initially chose the Microsoft Azure platform for these services and later added Amazon Web Services to avoid the risks of running its entire infrastructure onto a solution from a single vendor. The migration process, which took over two years, presented significant integration challenges. PostNL needed to move applications and data to the cloud, ensure the migrated applications continued communicating with the existing on-premises systems, and integrate various cloud environments.

Princess Polly, an Australian fashion boutique, was facing challenges in utilizing data effectively during a time of uncertainty. The company was preparing for a critical launch into the U.S. market and needed to support internal departments in making informed decisions. Anand Bhatt, the Head of Business Analytics, was tasked with building an analytics infrastructure that could demonstrate value quickly and efficiently. As the sole member of his team, Anand needed to maximize his time generating value for the business and minimize manual, time-consuming tasks. A key area of focus was cash flow analysis, with the aim of understanding which decisions were impacting the business’ bottom line to make more effective decisions.

Red Ventures (RV), a global company with a focus on positively impacting people’s lives and communities, was facing a challenge in managing marketing data efficiently. The company's Red Digital division provides end-to-end performance marketing services to help business-to-consumer (B2C) services providers attract new customers. To deliver greater value to clients, RV needed to use timely insights from data to reach the right consumers. However, maintaining each client’s data in a separate cloud environment and integrating each client’s data for machine learning predictions was proving to be a tedious and time-consuming task. Data engineers had to write custom scripts to ingest data for each client, which was not an efficient use of their time and skills.

Redwood Logistics, a third-party logistics and transportation management firm, was struggling with managing a complex reporting structure that relied on multiple siloed warehouses. The rapidly growing business needed a modern data stack that could support its mergers and acquisitions strategy, providing leadership with an accurate overview of business performance in near real time. The company was generating 500,000 data points per hour, which was a significant challenge to manage and process. The old system could only load data once a day and was prone to numerous daily failures, becoming a massive maintenance burden. Redwood was initially cautious about using Fivetran’s high-volume data replication because the team needed to understand how it interacted with their existing databases.

Schüttflix, a German logistics start-up, aimed to digitize the construction industry, a sector traditionally reliant on pen and paper. The company sought to disrupt local supply chains by connecting suppliers, carriers, and buyers through a digital B2B platform. The challenge was to enable data-driven decision-making to speed up transactions and reduce costs. Alexander Rupp, Head of Data and Business Intelligence, was tasked with building a modern data stack. He needed to identify connectors that could tap into key data sources quickly and reliably. The goal was to provide stakeholders with the best data to make informed decisions.

Sleeping Duck, an Australian mattress company, was facing the challenge of managing and deriving actionable insights from data scattered across various sources. The data resided in Software as a Service (SaaS) platforms, web apps, marketing platforms such as Facebook and Google Ads, and in the company’s own product. The process of extracting relevant information from these disparate sources was complex and manual. The company's engineers would have had to write and maintain custom scripts to extract data, a practice that was neither scalable nor sustainable. The company needed a solution that could efficiently pull in data from these sources, manage it, and feed it into their business intelligence solutions for data-driven decision making.

Snowflake, a leading data cloud company, was looking to centralize its data within the organization's Snowflake instance, ‘Snowhouse,’ to power segmentation models, recommendation engines, and ultimately build a 360-degree view of customers. The marketing intelligence team at Snowflake had a bold vision to predict real-time ROI to dynamically optimize all Snowflake marketing programs, disrupting legacy B2B marketing analytics practices, and create huge efficiencies. However, the company faced challenges in breaking down data silos and enabling efficient analytics. Snowflake used to keep its data modeling and transformation logic within a separate BI tool, which was time-consuming and prone to error. Every time the business needed to run models out of the tool, or conduct ad-hoc analytics, analysts needed to recreate their models from scratch.

SpotOn, a rapidly growing software and payment company, faced significant challenges in efficiently transforming their captured customer transaction data into fast, reliable, and informative reporting for their clients. As the company scaled, the complexity of turning data into reporting for customers and internal stakeholders increased, with client data scattered across 30 unconnected MySQL databases. The engineering team lacked a central repository for efficient reporting generation. The existing data transformation process using stored procedures in Snowflake became increasingly complex and resource-intensive, with over 2,000 lines of code behind a single table. Changes were not automatically monitored or logged without version control, making quality assurance time-consuming and scaling required writing code from scratch for each new use case. This resulted in high costs, resource-intensive processes, and suboptimal results, impacting the company's ability to scale quickly to meet growing customer needs.

Super Dispatch, an online platform for auto transport, was facing challenges in onboarding new users and optimizing experiences for active users. The company's data was decentralized and scattered across various digital properties, business systems, and marketing tools. Employees were relying on spreadsheets shared around the company for different purposes such as marketing, billing, and sales. The data was downloaded from business systems or Software as a Service (SaaS) platforms individually and analyzed in Excel. This posed a significant challenge for Aman Malhotra, a veteran in the marketing, sales, and operations analysis industry, who was hired to improve user activation, retention, and monetization through the use of data.

Swapfiets, the world’s first ‘bicycle-as-a-service’ company, was facing a challenge in understanding behaviors in the emerging market. The company's growth strategy relied on identifying new cities, winning new subscribers cost-effectively, and establishing an efficient local support network. However, the company was struggling with data management. The data engineering team had built custom Python scripts to extract data into their central Redshift instance, which was manageable when pulling from just a couple of data sources. However, as the business started to expand, this approach proved impractical. Swapfiets needed a more streamlined approach to data ingestion to make sense of critical subscription and usage data. It was crucial for Swapfiets to understand its target demographic and how best to provide local support, carefully target its marketing, and avoid over-provisioning stock.

Testing for All, a UK-based non-profit organisation, was launched to provide mass Covid-19 testing at a low cost. The organisation aimed to deliver 5,000 high-quality Covid-19 tests a day at half the price of other services. However, they faced a significant challenge in managing personal data, medical test results, and biological samples while maintaining a prompt and user-friendly service at scale. The process involved a six-step procedure, starting with registration and dispatching a test kit, and ending with receiving lab results. The organisation needed a privacy-centric technology stack that could handle the complexity of the process and ensure speed and efficiency in both the eCommerce part (signing up and ordering the kits) and the science part (the labs providing a range of swabbing techniques).

Untitled is a company that is building a platform to help its clients leverage data across departments. The company's data products enable non-technical staff to derive key insights, leading to increased revenue, decreased operating costs, and the development of sophisticated AI and ML capabilities. However, the traditional process of building data pipelines, which is crucial for transferring data from one point to another, was proving to be a significant challenge. This process was time-consuming, accounting for as much as 44% of data engineers’ time, and was hindering the rapid development of their platform.

Vida Health, a digital health company, was facing challenges with its data infrastructure. The company collects data on customers' medical history, past insurance claims, lab test results, and log data from health-tech devices to provide personalized virtual care. However, their custom-built solution using Python scripts and cron jobs to load and transform data in BigQuery was not scalable and often failed when data volume spiked. The pipeline was poorly documented and understood by only a few people on the data team, leading to reporting downtime of 2-3 days when issues arose. The company had recently consolidated its data engineering, data science, and data analytics functions into one team, aiming to improve collaboration. However, the existing data infrastructure was not reliable or accessible enough to best serve their customers and meet their goal of onboarding more than ten new clients in less than six months.

Wallbox, an electric vehicle charging and energy management company, faced a significant challenge in managing its data. Since its inception in 2015, the company experienced rapid growth, expanding from 50 to over 1,000 employees in a short span of time. This growth led to an increase in the number of tools and applications used across different departments, resulting in data silos that hindered insight and quality control. The company's data was scattered across various platforms, making it difficult to trace and resolve quality issues. Additionally, the business logic embedded in the dashboard was complex to evolve. Another challenge was the regular updating of tools required for custom integrations, which proved to be a costly and time-consuming process. Wallbox needed a solution to break these silos and consolidate all its data in a single, easily accessible location.

Westwing, a leading European eCommerce company, was facing challenges with its outdated technology stack and inefficient data architecture. The company recognized the importance of integrating data in a centralized location, but the manual work on their on-premise architecture was becoming increasingly time-consuming. To integrate with each different data source, every line of code had to be programmed with Python. This was slowing down the company's growth and preventing it from achieving a holistic view of the business. Westwing decided to move its architecture to the cloud, with Snowflake as the data warehouse, and outsource commodity services to focus on its strategic goal of scaling an eCommerce platform. However, with an ambitious cloud migration deadline looming, Westwing needed to find an ELT solution that could quickly and efficiently automate access to data.

WeWork, a global provider of flexible office spaces, faced the challenge of securely managing and leveraging its vast data resources to drive business decisions. As a publicly-traded company, WeWork had to meet stringent regulatory compliance requirements, ensuring data governance across disparate sources and silos. The company needed to track data movement and user access over time, demonstrating to auditors and regulators that customer information was safe from internal and external threats. The challenge was not only to pull in data and provide access but also to maintain a historical record of data ingestion, changes to the database, and access. The company also aimed to create a culture of data literacy and innovation, empowering business agility through the use of data.

World Fuel Services (WFS), a Fortune 150 company, faced significant challenges in managing and utilizing its data effectively. The company had grown through numerous acquisitions, each with its own client lists and data sources, making it difficult to gain a comprehensive view of customers across the entire organization. Additionally, the company's existing ETL pipelines pulled data in batches once a day into an on-premise Oracle database, which quickly became too large to run live queries effectively. The company also faced the challenge of managing data from dozens of ERP and billing information services across its subsidiaries, which was particularly critical during the global pandemic when the company needed to increase accounts receivable efforts to maintain revenue.

Yardzen, an online landscape design firm, was facing significant challenges in managing its data pipelines. The company's Data Engineering Lead, Andrea Kyrala, was tasked with integrating data from numerous SaaS tools and product databases into BigQuery, as well as establishing a flexible and secure data architecture. However, building custom pipelines to BigQuery in-house was a time-consuming process, often requiring weeks of work digging through API documentation. Moreover, the ETL pipelines were brittle and frequently required intensive maintenance. Analysts and marketers were manually exporting individual reports from each marketing platform to understand ad and marketing performance, a process that was not only painstaking and time-consuming but also made it difficult for leadership to gain a unified view of advertising spend and performance across platforms. Often, Andrea didn’t have the time for complex transformation and cleanup that would ultimately save the business analyst time in the backend.

YipitData, a trusted source of insights using alternative data, faced a significant challenge as it began to grow rapidly. The company's analytics activities were running on dozens of Amazon Redshift clusters, with each team within the company maintaining its own clusters. This arrangement became cumbersome, especially when YipitData needed to share common data sets across teams. The company's product teams analyze billions of data points each day to provide granular insights that drive the successful decision-making of hundreds of investment funds and highly innovative corporations. However, the existing system was slowing down analytics and making it difficult for the company to stay ahead of its market.

BizCover, Australia’s largest online business insurance provider, was facing a significant challenge in connecting data from various sources. The company's team of engineers had to build unique connectors using their own code, each requiring 40 to 80 hours of engineering time. This approach initially worked when connecting and syncing data from their database and Google Analytics. However, as the number of data sources increased, the task became overwhelming. BizCover needed to pull data from over 20 data sources into its centralized Snowflake data warehouse, with each source requiring its own connector. The company’s data engineers were managing this largely manual process, and BizCover needed to disseminate the insights they were gaining from the data across the core business more efficiently.

Blend, a fintech startup, was facing a significant challenge with its data ingestion process. Despite having adopted a modern data stack approach with Redshift at its core, getting data into and out of the data warehouse was proving to be a complex and time-consuming task. The process of pulling a single column from Salesforce or changing a field could take weeks, limiting access to time-critical data. The team was unable to prototype and rapidly iterate, and had to release straight to production to test their solutions, causing further complications for the operations team. As the company expanded, new tools like Asana, Marketo, and Lever were introduced to manage workflows and processes, each requiring data to be synced inside them to be effective. With the data engineering team’s limited bandwidth, they did not have the capacity to maintain a rapidly expanding list of SaaS platforms. This led to a decision point: commit to in-house tooling, or look for external providers.