Databricks Case Studies Real-Time Analytics at Scale: Akamai's Transformation with Delta Lake

Edit This Case Study Record

	Real-Time Analytics at Scale: Akamai's Transformation with Delta Lake Databricks

Real-Time Analytics at Scale: Akamai's Transformation with Delta Lake

Databricks

Technology Category	Cybersecurity & Privacy - Cloud Security Robots - Parallel Robots
Applicable Industries	National Security & Defense Retail
Applicable Functions	Sales & Marketing
Use Cases	Real-Time Location System (RTLS) Tamper Detection
Services	Cloud Planning, Design & Implementation Services Cybersecurity Services
Challenge	Akamai, a global content delivery network (CDN) provider, manages approximately 30% of the internet’s traffic through its 345,000 servers spread across more than 135 countries. In 2018, Akamai launched a web security analytics tool to provide its customers with a unified interface for assessing a wide range of streaming security events and perform real-time analysis. This tool ingests approximately 10GB of data related to security events per second, with data volumes increasing significantly during peak retail periods. The tool initially relied on an on-premises architecture running Apache Spark™ on Hadoop. However, Akamai faced challenges in meeting its strict service level agreements (SLAs) of 5 to 7 minutes from when an attack occurs until it is displayed in the tool. The company sought to improve ingestion and query speed to meet these SLAs and provide real-time data to its customers. Read More
About Customer	Akamai is a global content delivery network (CDN) provider that operates approximately 345,000 servers in more than 135 countries and over 1,300 networks worldwide. The company routes internet traffic for some of the largest enterprises in media, commerce, finance, retail, and many other industries. In addition to its CDN services, Akamai also provides cloud security solutions. In 2018, the company launched a web security analytics tool that offers its customers a single, unified interface for assessing a wide range of streaming security events and performing real-time analysis of those events. The tool helps Akamai's customers take informed actions in relation to security events in real time. Read More
Solution	After conducting proofs of concept with several companies, Akamai chose to base its streaming analytics architecture on Spark and the Databricks Lakehouse Platform. The web security analytics tool now ingests and transforms data, stores it in cloud storage, and sends the location of the file via Kafka. It then uses a Databricks Job as the ingest application. Delta Lake, the open source storage format at the base of the Databricks Lakehouse Platform, supports real-time querying on the web security analytics data and enables Akamai to scale quickly. Akamai also uses Databricks SQL (DBSQL) and Photon, which provide extremely fast query performance. The combination of Databricks’ streaming architecture, DBSQL, and Photon enables Akamai to achieve real-time analytics, translating to real-time business benefits. Read More Log in to view content
Contents

Technology Category

Cybersecurity & Privacy - Cloud Security

Robots - Parallel Robots

Applicable Industries

National Security & Defense

Retail

Applicable Functions

Sales & Marketing

Use Cases

Real-Time Location System (RTLS)

Tamper Detection

Services

Cloud Planning, Design & Implementation Services

Cybersecurity Services

Challenge

Akamai, a global content delivery network (CDN) provider, manages approximately 30% of the internet’s traffic through its 345,000 servers spread across more than 135 countries. In 2018, Akamai launched a web security analytics tool to provide its customers with a unified interface for assessing a wide range of streaming security events and perform real-time analysis. This tool ingests approximately 10GB of data related to security events per second, with data volumes increasing significantly during peak retail periods. The tool initially relied on an on-premises architecture running Apache Spark™ on Hadoop. However, Akamai faced challenges in meeting its strict service level agreements (SLAs) of 5 to 7 minutes from when an attack occurs until it is displayed in the tool. The company sought to improve ingestion and query speed to meet these SLAs and provide real-time data to its customers.

About Customer

Akamai is a global content delivery network (CDN) provider that operates approximately 345,000 servers in more than 135 countries and over 1,300 networks worldwide. The company routes internet traffic for some of the largest enterprises in media, commerce, finance, retail, and many other industries. In addition to its CDN services, Akamai also provides cloud security solutions. In 2018, the company launched a web security analytics tool that offers its customers a single, unified interface for assessing a wide range of streaming security events and performing real-time analysis of those events. The tool helps Akamai's customers take informed actions in relation to security events in real time.

Solution

After conducting proofs of concept with several companies, Akamai chose to base its streaming analytics architecture on Spark and the Databricks Lakehouse Platform. The web security analytics tool now ingests and transforms data, stores it in cloud storage, and sends the location of the file via Kafka. It then uses a Databricks Job as the ingest application. Delta Lake, the open source storage format at the base of the Databricks Lakehouse Platform, supports real-time querying on the web security analytics data and enables Akamai to scale quickly. Akamai also uses Databricks SQL (DBSQL) and Photon, which provide extremely fast query performance. The combination of Databricks’ streaming architecture, DBSQL, and Photon enables Akamai to achieve real-time analytics, translating to real-time business benefits.

Impact #1	The transition to the Databricks Lakehouse Platform has significantly improved Akamai's ability to meet its strict SLAs and provide real-time data to its customers. The use of Delta Lake, an open-source storage format, has not only improved query performance but also enabled Akamai to scale quickly in response to an 80% increase in traffic and data over the past year. The use of Databricks SQL and Photon has further enhanced query performance, enabling Akamai to provide real-time analytics. The move to Databricks has also improved customer experience, with over 85% of queries now completing in under 7 seconds. This real-time data provision helps Akamai's customers stay vigilant and maintain optimal security configurations.

Impact #1

The transition to the Databricks Lakehouse Platform has significantly improved Akamai's ability to meet its strict SLAs and provide real-time data to its customers. The use of Delta Lake, an open-source storage format, has not only improved query performance but also enabled Akamai to scale quickly in response to an 80% increase in traffic and data over the past year. The use of Databricks SQL and Photon has further enhanced query performance, enabling Akamai to provide real-time analytics. The move to Databricks has also improved customer experience, with over 85% of queries now completing in under 7 seconds. This real-time data provision helps Akamai's customers stay vigilant and maintain optimal security configurations.

Benefit #1	Min ingestion time reduced from 15 minutes to under 1 minute
Benefit #2	Over 85% of queries have a response time of 7 seconds or less
Benefit #3	70% of security event data has been moved from on-prem architecture to Databricks

Benefit #1

Min ingestion time reduced from 15 minutes to under 1 minute

Benefit #2

Over 85% of queries have a response time of 7 seconds or less

Benefit #3

70% of security event data has been moved from on-prem architecture to Databricks

Download PDF Version

Overview

Real-Time Analytics at Scale: Akamai's Transformation with Delta Lake

Operational Impact

Quantitative Benefit