ClickHouse Case Studies ClickHouse: Powering Darwinium's Security and Fraud Analytics

Edit This Case Study Record

	ClickHouse: Powering Darwinium's Security and Fraud Analytics ClickHouse

ClickHouse: Powering Darwinium's Security and Fraud Analytics

ClickHouse

Technology Category	Application Infrastructure & Middleware - Database Management & Storage Infrastructure as a Service (IaaS) - Cloud Databases
Applicable Industries	National Security & Defense Oil & Gas
Applicable Functions	Logistics & Transportation Product Research & Development
Use Cases	Cybersecurity Fraud Detection
Services	Cloud Planning, Design & Implementation Services System Integration
Challenge	Darwinium, a digital risk platform, was facing several challenges in the security and fraud domain. The platform needed to ingest and process data at a high throughput, deal with large volumes of data, and have capabilities to analyze data in a complex way. The database backend needed to handle high-speed writes and serve data for analysis as soon as it was ingested. Darwinium's real-time engine continuously profiles and monitors a digital asset, resulting in large volumes of data. The database needed to be capable of analyzing data at scale, and potentially process an entire year's worth of data. Technical types of fraud and security challenges required storing most digital datapoints for future investigations. The nature of analyzing fraudulent data required complex interactive analysis, and a database system that could respond in timeframes of 1 second or less, while providing a feature-rich functional toolbox. Read More
About Customer	Darwinium is a digital risk platform that supports real-time journey orchestration and continuous adaptive trust for digital user authentication. It was built to tackle complex business problems as they happen, adapting to adversaries regardless of how quickly they attack. The platform is designed for developers and data scientists to test, model, and deploy with ease, regardless of business processes or organizational constraints. Darwinium integrates with your Content Delivery Network (CDN) or Proxy as a Darwinium-hosted solution, or optional one-click install onto a new or existing Kubernetes cluster. Read More
Solution	Darwinium chose ClickHouse as their database engine after assessing several solutions. ClickHouse's mutable engine simplified the data ingestion pipeline complexity, and its choice of table engine led to simpler data pipelines. ClickHouse could easily handle a few thousand writes per second, with multiple data pipeline writers writing to a single ClickHouse server at any given instant. It also supported complex data types, which was a fundamental requirement for Darwinium to build upon for interactive analytics. ClickHouse's recent addition of JSON type support further met the analytics requirements of Darwinium. ClickHouse also supported the concept of tumbling data retention windows, where “hot” data could be initially placed on a fast access medium like a local SSD, with the ability to subsequently move the data to a relatively slower but cheaper storage system like S3. ClickHouse was also cloud-native, and could be run from a low-end laptop, an on-premise cluster of nodes, or on any of the myriad hardware configurations possible even on a single cloud provider like AWS. Read More Log in to view content
Contents

Technology Category

Application Infrastructure & Middleware - Database Management & Storage

Infrastructure as a Service (IaaS) - Cloud Databases

Applicable Industries

National Security & Defense

Oil & Gas

Applicable Functions

Logistics & Transportation

Product Research & Development

Use Cases

Cybersecurity

Fraud Detection

Services

Cloud Planning, Design & Implementation Services

System Integration

Challenge

Darwinium, a digital risk platform, was facing several challenges in the security and fraud domain. The platform needed to ingest and process data at a high throughput, deal with large volumes of data, and have capabilities to analyze data in a complex way. The database backend needed to handle high-speed writes and serve data for analysis as soon as it was ingested. Darwinium's real-time engine continuously profiles and monitors a digital asset, resulting in large volumes of data. The database needed to be capable of analyzing data at scale, and potentially process an entire year's worth of data. Technical types of fraud and security challenges required storing most digital datapoints for future investigations. The nature of analyzing fraudulent data required complex interactive analysis, and a database system that could respond in timeframes of 1 second or less, while providing a feature-rich functional toolbox.

About Customer

Darwinium is a digital risk platform that supports real-time journey orchestration and continuous adaptive trust for digital user authentication. It was built to tackle complex business problems as they happen, adapting to adversaries regardless of how quickly they attack. The platform is designed for developers and data scientists to test, model, and deploy with ease, regardless of business processes or organizational constraints. Darwinium integrates with your Content Delivery Network (CDN) or Proxy as a Darwinium-hosted solution, or optional one-click install onto a new or existing Kubernetes cluster.

Solution

Darwinium chose ClickHouse as their database engine after assessing several solutions. ClickHouse's mutable engine simplified the data ingestion pipeline complexity, and its choice of table engine led to simpler data pipelines. ClickHouse could easily handle a few thousand writes per second, with multiple data pipeline writers writing to a single ClickHouse server at any given instant. It also supported complex data types, which was a fundamental requirement for Darwinium to build upon for interactive analytics. ClickHouse's recent addition of JSON type support further met the analytics requirements of Darwinium. ClickHouse also supported the concept of tumbling data retention windows, where “hot” data could be initially placed on a fast access medium like a local SSD, with the ability to subsequently move the data to a relatively slower but cheaper storage system like S3. ClickHouse was also cloud-native, and could be run from a low-end laptop, an on-premise cluster of nodes, or on any of the myriad hardware configurations possible even on a single cloud provider like AWS.

Impact #1	ClickHouse's mutable engine simplified the data ingestion pipeline complexity, and its choice of table engine led to simpler data pipelines. It also supported complex data types, which was a fundamental requirement for Darwinium to build upon for interactive analytics. ClickHouse's recent addition of JSON type support further met the analytics requirements of Darwinium. ClickHouse also supported the concept of tumbling data retention windows, where “hot” data could be initially placed on a fast access medium like a local SSD, with the ability to subsequently move the data to a relatively slower but cheaper storage system like S3. ClickHouse was also cloud-native, and could be run from a low-end laptop, an on-premise cluster of nodes, or on any of the myriad hardware configurations possible even on a single cloud provider like AWS. This made ClickHouse a cloud native offering for Darwinium use cases.

Impact #1

ClickHouse's mutable engine simplified the data ingestion pipeline complexity, and its choice of table engine led to simpler data pipelines. It also supported complex data types, which was a fundamental requirement for Darwinium to build upon for interactive analytics. ClickHouse's recent addition of JSON type support further met the analytics requirements of Darwinium. ClickHouse also supported the concept of tumbling data retention windows, where “hot” data could be initially placed on a fast access medium like a local SSD, with the ability to subsequently move the data to a relatively slower but cheaper storage system like S3. ClickHouse was also cloud-native, and could be run from a low-end laptop, an on-premise cluster of nodes, or on any of the myriad hardware configurations possible even on a single cloud provider like AWS. This made ClickHouse a cloud native offering for Darwinium use cases.

Benefit #1	ClickHouse could easily handle a few thousand writes per second, with multiple data pipeline writers writing to a single ClickHouse server at any given instant.
Benefit #2	ClickHouse server side metrics showed a consistent and very low (<5%) user space and system space core usage while the write operations alone were being executed.
Benefit #3	ClickHouse can utilise all of the cores available on a single node that it is running on to process a query.

Benefit #1

ClickHouse could easily handle a few thousand writes per second, with multiple data pipeline writers writing to a single ClickHouse server at any given instant.

Benefit #2

ClickHouse server side metrics showed a consistent and very low (<5%) user space and system space core usage while the write operations alone were being executed.

Benefit #3

ClickHouse can utilise all of the cores available on a single node that it is running on to process a query.

Download PDF Version

Overview

ClickHouse: Powering Darwinium's Security and Fraud Analytics

Operational Impact

Quantitative Benefit