Case Studies How we built BI on Clickhouse with row-level security in Deutsche Bank Technology Centre

Edit This Case Study Record

	How we built BI on Clickhouse with row-level security in Deutsche Bank Technology Centre

How we built BI on Clickhouse with row-level security in Deutsche Bank Technology Centre

Technology Category	Analytics & Modeling - Big Data Analytics Analytics & Modeling - Predictive Analytics Application Infrastructure & Middleware - Data Exchange & Integration Application Infrastructure & Middleware - Database Management & Storage
Applicable Industries	Finance & Insurance
Applicable Functions	Business Operation Quality Assurance
Services	Data Science Services Software Design & Engineering Services System Integration
Challenge	Deutsche Bank Technology Centre faced significant challenges in managing and analyzing data within its Investment Bank division. The natural data silos and the need for a robust Business Intelligence (BI) system were evident. The existing data warehouse solutions were either too expensive or too slow, such as Vertica and Hive, respectively. Additionally, the bank required a data-driven access control mechanism that could provide record-level granularity and full access to SQL, while reusing existing bank-wide access rules. The challenge was to find a solution that could handle heterogeneous data from 72 different data sources and systems, manage over 100 ETL jobs, and provide a seamless user experience across various UIs. Read More
About Customer	Deutsche Bank Technology Centre is a part of Deutsche Bank, one of the world's leading financial service providers. The Technology Centre focuses on developing and implementing cutting-edge technology solutions to support the bank's operations and services. With a global presence and a diverse range of financial products, Deutsche Bank serves millions of customers worldwide. The Technology Centre plays a crucial role in ensuring the bank's technological infrastructure is robust, secure, and capable of handling the complex demands of the financial industry. The centre is responsible for managing data, developing software solutions, and ensuring compliance with regulatory requirements. Read More
Solution	To address these challenges, Deutsche Bank Technology Centre implemented a combination of Spark, ClickHouse, and Tableau/RShiny for their BI needs. In 2017, they adopted Spark for its powerful data processing capabilities and ClickHouse for its high-performance columnar storage. Tableau and RShiny were used for data visualization and reporting. By 2018, the solution evolved to include Alpakka for data integration, Kafka for real-time data streaming, and a Web UI for user interaction. The Access-Based Access Control (ABAC) system was integrated to provide data-driven access control with record-level granularity. This comprehensive solution allowed the bank to track changes, manage investment planning, and improve client interaction quality. The use of ClickHouse enabled fast query performance, while Spark and Kafka ensured efficient data processing and real-time analytics. Read More Log in to view content
Contents

Technology Category

Analytics & Modeling - Big Data Analytics

Analytics & Modeling - Predictive Analytics

Application Infrastructure & Middleware - Data Exchange & Integration

Application Infrastructure & Middleware - Database Management & Storage

Applicable Industries

Finance & Insurance

Applicable Functions

Business Operation

Quality Assurance

Services

Data Science Services

Software Design & Engineering Services

System Integration

Challenge

Deutsche Bank Technology Centre faced significant challenges in managing and analyzing data within its Investment Bank division. The natural data silos and the need for a robust Business Intelligence (BI) system were evident. The existing data warehouse solutions were either too expensive or too slow, such as Vertica and Hive, respectively. Additionally, the bank required a data-driven access control mechanism that could provide record-level granularity and full access to SQL, while reusing existing bank-wide access rules. The challenge was to find a solution that could handle heterogeneous data from 72 different data sources and systems, manage over 100 ETL jobs, and provide a seamless user experience across various UIs.

About Customer

Deutsche Bank Technology Centre is a part of Deutsche Bank, one of the world's leading financial service providers. The Technology Centre focuses on developing and implementing cutting-edge technology solutions to support the bank's operations and services. With a global presence and a diverse range of financial products, Deutsche Bank serves millions of customers worldwide. The Technology Centre plays a crucial role in ensuring the bank's technological infrastructure is robust, secure, and capable of handling the complex demands of the financial industry. The centre is responsible for managing data, developing software solutions, and ensuring compliance with regulatory requirements.

Solution

To address these challenges, Deutsche Bank Technology Centre implemented a combination of Spark, ClickHouse, and Tableau/RShiny for their BI needs. In 2017, they adopted Spark for its powerful data processing capabilities and ClickHouse for its high-performance columnar storage. Tableau and RShiny were used for data visualization and reporting. By 2018, the solution evolved to include Alpakka for data integration, Kafka for real-time data streaming, and a Web UI for user interaction. The Access-Based Access Control (ABAC) system was integrated to provide data-driven access control with record-level granularity. This comprehensive solution allowed the bank to track changes, manage investment planning, and improve client interaction quality. The use of ClickHouse enabled fast query performance, while Spark and Kafka ensured efficient data processing and real-time analytics.

Impact #1	The implementation of ClickHouse and Spark significantly improved the bank's ability to manage and analyze data from multiple sources.
Impact #2	The integration of ABAC provided a robust security mechanism, ensuring data-driven access control with record-level granularity.
Impact #3	The use of Tableau and RShiny for data visualization enhanced the reporting capabilities, making it easier for users to interact with the data.

Impact #1

The implementation of ClickHouse and Spark significantly improved the bank's ability to manage and analyze data from multiple sources.

Impact #2

The integration of ABAC provided a robust security mechanism, ensuring data-driven access control with record-level granularity.

Impact #3

The use of Tableau and RShiny for data visualization enhanced the reporting capabilities, making it easier for users to interact with the data.

Benefit #1	Managed data from 72 different data sources and systems.
Benefit #2	Handled over 100 ETL jobs efficiently.
Benefit #3	Utilized 4 servers with a total storage capacity of 500GB/26TB.

Benefit #1

Managed data from 72 different data sources and systems.

Benefit #2

Handled over 100 ETL jobs efficiently.

Benefit #3

Utilized 4 servers with a total storage capacity of 500GB/26TB.

Download PDF Version

Overview

How we built BI on Clickhouse with row-level security in Deutsche Bank Technology Centre

Operational Impact

Quantitative Benefit