Neptune.ai Case Studies Brainly's Integration with Amazon SageMaker and Neptune for Enhanced Machine Learning Capabilities

Edit This Case Study Record

	Brainly's Integration with Amazon SageMaker and Neptune for Enhanced Machine Learning Capabilities Neptune.ai

Brainly's Integration with Amazon SageMaker and Neptune for Enhanced Machine Learning Capabilities

Neptune.ai

Technology Category	Analytics & Modeling - Machine Learning Infrastructure as a Service (IaaS) - Cloud Middleware & Microservices
Applicable Industries	Education Equipment & Machinery
Use Cases	Behavior & Emotion Tracking Virtual Training
Services	System Integration Training
Challenge	Brainly, a leading global learning platform, faced a challenge with their machine learning-powered feature, Snap to Solve. The feature allows users to upload a photo of a problem, which the system then detects and provides solutions for. The Visual Search team, responsible for the Visual Content Extraction (VICE) system of Snap to Solve, used Amazon SageMaker to run their computing workloads and serve their models. However, as the number of training runs on their large compute architectures increased, they found that their logs from Amazon SageMaker needed to be trackable and manageable to avoid workflow bottlenecks. They needed a tool that could scale regardless of the experiment volume. While they tried using SageMaker Experiments for tracking, they found the tracking UX and Python client unsatisfactory. Read More
About Customer	Brainly is a leading learning platform worldwide, providing an extensive Knowledge Base for all school subjects and grades. Each month, over 350 million students, parents, and educators rely on Brainly as a trusted platform to accelerate understanding and learning. One of their core features is Snap to Solve, a machine learning-powered feature that allows users to upload a photo of a problem for the system to detect and provide solutions. The Visual Search team at Brainly is responsible for the Visual Content Extraction (VICE) system of the Snap to Solve product. Read More
Solution	Brainly's team found a solution in Neptune, a tool that could integrate well with their technology stack and their CD pipeline. They developed a custom template to connect Neptune with Amazon SageMaker Pipelines using the `NEPTUNE_CUSTOM_RUN_ID` feature. This allowed them to log and retrieve experiment metadata and usage metrics for the entire computational pipeline in a single run, ensuring efficient organization and easy reproduction of their experiments. Neptune's intuitive UI allowed anyone in the team or organization to easily analyze the tracked data. Additionally, Neptune's unlimited user feature and collaboration features such as sharing UI views through persistent links, enabled better visibility across the organization. The team also used Neptune to monitor and optimize their computational resource consumption, particularly for their multi-GPU training and large data processing jobs. Read More Log in to view content
Contents

Technology Category

Analytics & Modeling - Machine Learning

Infrastructure as a Service (IaaS) - Cloud Middleware & Microservices

Applicable Industries

Education

Equipment & Machinery

Use Cases

Behavior & Emotion Tracking

Virtual Training

Services

System Integration

Training

Challenge

Brainly, a leading global learning platform, faced a challenge with their machine learning-powered feature, Snap to Solve. The feature allows users to upload a photo of a problem, which the system then detects and provides solutions for. The Visual Search team, responsible for the Visual Content Extraction (VICE) system of Snap to Solve, used Amazon SageMaker to run their computing workloads and serve their models. However, as the number of training runs on their large compute architectures increased, they found that their logs from Amazon SageMaker needed to be trackable and manageable to avoid workflow bottlenecks. They needed a tool that could scale regardless of the experiment volume. While they tried using SageMaker Experiments for tracking, they found the tracking UX and Python client unsatisfactory.

About Customer

Brainly is a leading learning platform worldwide, providing an extensive Knowledge Base for all school subjects and grades. Each month, over 350 million students, parents, and educators rely on Brainly as a trusted platform to accelerate understanding and learning. One of their core features is Snap to Solve, a machine learning-powered feature that allows users to upload a photo of a problem for the system to detect and provide solutions. The Visual Search team at Brainly is responsible for the Visual Content Extraction (VICE) system of the Snap to Solve product.

Solution

Brainly's team found a solution in Neptune, a tool that could integrate well with their technology stack and their CD pipeline. They developed a custom template to connect Neptune with Amazon SageMaker Pipelines using the `NEPTUNE_CUSTOM_RUN_ID` feature. This allowed them to log and retrieve experiment metadata and usage metrics for the entire computational pipeline in a single run, ensuring efficient organization and easy reproduction of their experiments. Neptune's intuitive UI allowed anyone in the team or organization to easily analyze the tracked data. Additionally, Neptune's unlimited user feature and collaboration features such as sharing UI views through persistent links, enabled better visibility across the organization. The team also used Neptune to monitor and optimize their computational resource consumption, particularly for their multi-GPU training and large data processing jobs.

Impact #1	The integration of Neptune with Brainly's technology stack and CD pipeline resulted in significant operational improvements. The team was able to efficiently organize their work and easily reproduce their experiments from SageMaker Pipelines. The intuitive UI of Neptune allowed anyone in the team or organization to easily analyze the tracked data, fostering a more collaborative and inclusive work environment. Furthermore, Neptune's monitoring features played a crucial role in maximizing the team’s GPU usage, providing deeper insights into resource utilization than what was provided by a Cloud vendor. This allowed the team to optimize their data augmentation jobs, improving data processing speed and overall efficiency.

Impact #1

The integration of Neptune with Brainly's technology stack and CD pipeline resulted in significant operational improvements. The team was able to efficiently organize their work and easily reproduce their experiments from SageMaker Pipelines. The intuitive UI of Neptune allowed anyone in the team or organization to easily analyze the tracked data, fostering a more collaborative and inclusive work environment. Furthermore, Neptune's monitoring features played a crucial role in maximizing the team’s GPU usage, providing deeper insights into resource utilization than what was provided by a Cloud vendor. This allowed the team to optimize their data augmentation jobs, improving data processing speed and overall efficiency.

Benefit #1	Neptune's integration allowed for the logging and retrieval of experiment metadata and usage metrics for the entire computational pipeline in a single run.
Benefit #2	Neptune's unlimited user feature and collaboration features enabled better visibility across the organization.
Benefit #3	Neptune's monitoring features helped in optimizing computational resource consumption, particularly for multi-GPU training and large data processing jobs.

Benefit #1

Neptune's integration allowed for the logging and retrieval of experiment metadata and usage metrics for the entire computational pipeline in a single run.

Benefit #2

Neptune's unlimited user feature and collaboration features enabled better visibility across the organization.

Benefit #3

Neptune's monitoring features helped in optimizing computational resource consumption, particularly for multi-GPU training and large data processing jobs.

Download PDF Version

Overview

Brainly's Integration with Amazon SageMaker and Neptune for Enhanced Machine Learning Capabilities

Operational Impact

Quantitative Benefit