ClickHouse Case Studies Building a Unified Data Platform with ClickHouse: A Case Study on Synq

Edit This Case Study Record

	Building a Unified Data Platform with ClickHouse: A Case Study on Synq ClickHouse

Building a Unified Data Platform with ClickHouse: A Case Study on Synq

ClickHouse

Technology Category	Analytics & Modeling - Machine Learning Platform as a Service (PaaS) - Application Development Platforms
Applicable Industries	Buildings Cement
Applicable Functions	Product Research & Development Quality Assurance
Use Cases	Building Automation & Control Time Sensitive Networking
Services	System Integration Testing & Certification
Challenge	Synq, a data observability platform, faced the challenge of managing the complexity, variety, and increasing volumes of data that powered their software system. The company needed to merge operational and analytical needs into a unified data platform. They were dealing with a continuous stream of data from dozens of systems, with frequent bursts of volume when customers ran large batch processing jobs or when new customers were onboarded. The company had set ambitious performance goals for backfilling data and wanted to provide immediate value to customers as they onboarded their product. They also wanted an infrastructure that could serve their first set of defined use cases and provide functionality to support new use cases quickly. Lastly, they aimed to build a single platform that could store their raw log data and act as a serving layer for most data use cases needed by their applications and APIs. Read More
About Customer	Synq is a data observability platform that analyzes log-level data from complex data ecosystems. It is a large-scale log processing engine that ingests and processes data from dozens of systems. The platform is designed to provide deep integration into ClickHouse clusters with capabilities to detect delayed data loads and uncover hidden data anomalies. It also offers automatically created data lineage and tooling for managing data quality. Synq serves teams at companies such as Typeform, Instabee, and LendInvest, helping them monitor their cloud data stacks. Read More
Solution	Synq found the solution to their challenges in ClickHouse, a high-performance column-oriented database management system. After a few days of testing, they found that ClickHouse could ingest tens of thousands of rows per second, create query-specific data models, and maintain consistent read query performance under heavy ingest load. To focus their entire development team on the R&D of their platform, they partnered with ClickHouse Cloud. They built a solid ingestion system using the officially maintained Go client clickhouse-go. They also leveraged the ReplacingMergeTree table engine to handle duplicate events. To optimize performance, they created specialized tables that transformed their raw logs data to a format optimized for their queries. They also used the popular data transformation framework dbt to create auditing tables that extract summary statistics about their log data. Finally, they used their ClickHouse cluster as a backbone for many other use cases, including in-app analytics. Read More Log in to view content
Contents

Technology Category

Analytics & Modeling - Machine Learning

Platform as a Service (PaaS) - Application Development Platforms

Applicable Industries

Buildings

Cement

Applicable Functions

Product Research & Development

Quality Assurance

Use Cases

Building Automation & Control

Time Sensitive Networking

Services

System Integration

Testing & Certification

Challenge

Synq, a data observability platform, faced the challenge of managing the complexity, variety, and increasing volumes of data that powered their software system. The company needed to merge operational and analytical needs into a unified data platform. They were dealing with a continuous stream of data from dozens of systems, with frequent bursts of volume when customers ran large batch processing jobs or when new customers were onboarded. The company had set ambitious performance goals for backfilling data and wanted to provide immediate value to customers as they onboarded their product. They also wanted an infrastructure that could serve their first set of defined use cases and provide functionality to support new use cases quickly. Lastly, they aimed to build a single platform that could store their raw log data and act as a serving layer for most data use cases needed by their applications and APIs.

About Customer

Synq is a data observability platform that analyzes log-level data from complex data ecosystems. It is a large-scale log processing engine that ingests and processes data from dozens of systems. The platform is designed to provide deep integration into ClickHouse clusters with capabilities to detect delayed data loads and uncover hidden data anomalies. It also offers automatically created data lineage and tooling for managing data quality. Synq serves teams at companies such as Typeform, Instabee, and LendInvest, helping them monitor their cloud data stacks.

Solution

Synq found the solution to their challenges in ClickHouse, a high-performance column-oriented database management system. After a few days of testing, they found that ClickHouse could ingest tens of thousands of rows per second, create query-specific data models, and maintain consistent read query performance under heavy ingest load. To focus their entire development team on the R&D of their platform, they partnered with ClickHouse Cloud. They built a solid ingestion system using the officially maintained Go client clickhouse-go. They also leveraged the ReplacingMergeTree table engine to handle duplicate events. To optimize performance, they created specialized tables that transformed their raw logs data to a format optimized for their queries. They also used the popular data transformation framework dbt to create auditing tables that extract summary statistics about their log data. Finally, they used their ClickHouse cluster as a backbone for many other use cases, including in-app analytics.

Impact #1	The use of ClickHouse has allowed Synq to fully merge their operations and analytics storage, enabling them to think about their system in terms of use cases, knowing that they have a performant data platform and other necessary building blocks to support them. The ability to control underlying storage engines, ingest mechanics, or query settings has given them extreme control over their storage, which has so far handled any use case they had in mind with performance that can support interactive user-facing experiences. Materialized views and integration with dbt have given them a lot of flexibility to quickly develop new data use cases without any extra ETL code or large migrations. This has made development extremely efficient and has allowed them to provide immediate value to their customers.

Impact #1

The use of ClickHouse has allowed Synq to fully merge their operations and analytics storage, enabling them to think about their system in terms of use cases, knowing that they have a performant data platform and other necessary building blocks to support them. The ability to control underlying storage engines, ingest mechanics, or query settings has given them extreme control over their storage, which has so far handled any use case they had in mind with performance that can support interactive user-facing experiences. Materialized views and integration with dbt have given them a lot of flexibility to quickly develop new data use cases without any extra ETL code or large migrations. This has made development extremely efficient and has allowed them to provide immediate value to their customers.

Benefit #1	ClickHouse could ingest tens of thousands of rows per second
Benefit #2	Maintained consistent read query performance under heavy ingest load
Benefit #3	Optimized complex analytical queries down to <100ms milliseconds latency

Benefit #1

ClickHouse could ingest tens of thousands of rows per second

Benefit #2

Maintained consistent read query performance under heavy ingest load

Benefit #3

Optimized complex analytical queries down to <100ms milliseconds latency

Download PDF Version

Overview

Building a Unified Data Platform with ClickHouse: A Case Study on Synq

Operational Impact

Quantitative Benefit