Storyblocks' Transformation: Redesigning Data Pipeline with Confluent

To manage these challenges, Storyblocks needed a new solution that could form the backbone of an entirely new data pipeline. They began to use Apache Kafka® and did a proof of concept in conjunction with using a schema registry from Aiven. However, they weren't getting the support for Kafka that they needed from Aiven, so they started to use Confluent Cloud, a fully managed cloud service for Kafka. They also began to spread the use of Kafka as an event bus for more streaming applications and machine learning (ML) features. With this Confluent-backed data pipeline in place, the Storyblocks team could begin to affect a true digital transformation both internally and externally. Instead of implementing a queue for inter-services communication, the team just puts it on the pipeline where it’s stored forever. This sort of infinite storage is powerful for two reasons: Events can be replayed on demand with powerful in-built schema validation and analysts can look at historical data when answering various questions for the business.