ClickHouse Case Studies HIFI's Transition from BigQuery to ClickHouse for Enhanced Music Royalty Data Management

Edit This Case Study Record

	HIFI's Transition from BigQuery to ClickHouse for Enhanced Music Royalty Data Management ClickHouse

HIFI's Transition from BigQuery to ClickHouse for Enhanced Music Royalty Data Management

ClickHouse

Technology Category	Application Infrastructure & Middleware - Database Management & Storage Application Infrastructure & Middleware - Middleware, SDKs & Libraries
Applicable Industries	Buildings Cement
Applicable Functions	Procurement
Use Cases	Inventory Management Retail Store Automation
Services	Cloud Planning, Design & Implementation Services System Integration
Challenge	HIFI, a company providing financial and business insights to music creators, was facing challenges with its data management system. The company ingests a significant amount of royalty data, with a single HIFI Enterprise account having over half a gigabyte of associated royalty data representing over 25 million rows of streaming and other transaction data. This data needs to load into the user interface as soon as a customer logs in, and there can be multiple customers logging in simultaneously. Previously, it could take up to 30 seconds to load the data, and sometimes it would not load at all due to timeouts. HIFI was using Google Cloud's BigQuery (BQ) to store royalty data, but the pricing structure of BQ was a major challenge. It discouraged data usage and contradicted HIFI's data-driven values. Google's solution to purchase BQ slots ahead of time was not feasible for HIFI as a startup, as usage patterns could change dramatically week to week. Read More
About Customer	HIFI is a company that is building innovative technologies to provide critical financial and business insights for its members, primarily music creators. The company offers a product called HIFI Enterprise, which brings intelligent revenue processing to sophisticated businesses handling music royalty data at scale. This includes companies and funds acquiring music rights and business managers who work with the most notable artists in the world. HIFI ingests a lot of royalty data, with a single HIFI Enterprise account having over half a gigabyte of associated royalty data representing over 25 million rows of streaming and other transaction data. The company is backed by top VCs and chart-topping creators across the globe. Read More
Solution	HIFI decided to switch to ClickHouse, a fast open-source column-oriented database management system that allows for generating analytical data reports in real-time using SQL queries. ClickHouse's performance exceeded all other column-oriented database management systems, processing billions of rows and tens of gigabytes of data per server per second. With ClickHouse, even the largest datasets loaded in under a few seconds, improving the user experience significantly. ClickHouse also proved beneficial for calculating internal metrics such as total royalties in the system and indicators like total royalties paid out in the last year. However, the speed of ClickHouse came with some surprises, like operations like joins not working as expected. To overcome this, HIFI used a JOIN engine table to update records more efficiently. HIFI also connected ClickHouse to PostgreSQL databases for non-royalty data such as customer account data and metadata. Read More Log in to view content
Contents

Technology Category

Application Infrastructure & Middleware - Database Management & Storage

Application Infrastructure & Middleware - Middleware, SDKs & Libraries

Applicable Industries

Buildings

Cement

Applicable Functions

Procurement

Use Cases

Inventory Management

Retail Store Automation

Services

Cloud Planning, Design & Implementation Services

System Integration

Challenge

HIFI, a company providing financial and business insights to music creators, was facing challenges with its data management system. The company ingests a significant amount of royalty data, with a single HIFI Enterprise account having over half a gigabyte of associated royalty data representing over 25 million rows of streaming and other transaction data. This data needs to load into the user interface as soon as a customer logs in, and there can be multiple customers logging in simultaneously. Previously, it could take up to 30 seconds to load the data, and sometimes it would not load at all due to timeouts. HIFI was using Google Cloud's BigQuery (BQ) to store royalty data, but the pricing structure of BQ was a major challenge. It discouraged data usage and contradicted HIFI's data-driven values. Google's solution to purchase BQ slots ahead of time was not feasible for HIFI as a startup, as usage patterns could change dramatically week to week.

About Customer

HIFI is a company that is building innovative technologies to provide critical financial and business insights for its members, primarily music creators. The company offers a product called HIFI Enterprise, which brings intelligent revenue processing to sophisticated businesses handling music royalty data at scale. This includes companies and funds acquiring music rights and business managers who work with the most notable artists in the world. HIFI ingests a lot of royalty data, with a single HIFI Enterprise account having over half a gigabyte of associated royalty data representing over 25 million rows of streaming and other transaction data. The company is backed by top VCs and chart-topping creators across the globe.

Solution

HIFI decided to switch to ClickHouse, a fast open-source column-oriented database management system that allows for generating analytical data reports in real-time using SQL queries. ClickHouse's performance exceeded all other column-oriented database management systems, processing billions of rows and tens of gigabytes of data per server per second. With ClickHouse, even the largest datasets loaded in under a few seconds, improving the user experience significantly. ClickHouse also proved beneficial for calculating internal metrics such as total royalties in the system and indicators like total royalties paid out in the last year. However, the speed of ClickHouse came with some surprises, like operations like joins not working as expected. To overcome this, HIFI used a JOIN engine table to update records more efficiently. HIFI also connected ClickHouse to PostgreSQL databases for non-royalty data such as customer account data and metadata.

Impact #1	The transition to ClickHouse has significantly improved HIFI's operational efficiency. The speed of ClickHouse has not only enhanced the user experience but also boosted the company's confidence in its ability to scale as it adds more customers and rolls out new products. The company can now calculate internal metrics such as total royalties in the system and indicators like total royalties paid out in the last year more quickly and accurately. Moreover, the company has overcome the challenges of joins in ClickHouse by using a JOIN engine table, which has made record updating more efficient. HIFI also successfully connected ClickHouse to PostgreSQL databases for non-royalty data, further enhancing its data management capabilities.

Impact #1

The transition to ClickHouse has significantly improved HIFI's operational efficiency. The speed of ClickHouse has not only enhanced the user experience but also boosted the company's confidence in its ability to scale as it adds more customers and rolls out new products. The company can now calculate internal metrics such as total royalties in the system and indicators like total royalties paid out in the last year more quickly and accurately. Moreover, the company has overcome the challenges of joins in ClickHouse by using a JOIN engine table, which has made record updating more efficient. HIFI also successfully connected ClickHouse to PostgreSQL databases for non-royalty data, further enhancing its data management capabilities.

Benefit #1	Data load time reduced from 30 seconds to a few seconds
Benefit #2	Ability to process billions of rows and tens of gigabytes of data per server per second
Benefit #3	Efficient updating of records using JOIN engine table

Benefit #1

Data load time reduced from 30 seconds to a few seconds

Benefit #2

Ability to process billions of rows and tens of gigabytes of data per server per second

Benefit #3

Efficient updating of records using JOIN engine table

Download PDF Version

Overview

HIFI's Transition from BigQuery to ClickHouse for Enhanced Music Royalty Data Management

Operational Impact

Quantitative Benefit