Neptune.ai

Waabi, a company focused on developing the next generation of self-driving truck technology, faced a significant challenge in managing their large-scale experimentation workflow. Their Machine Learning teams, organized around different technical pillars, constantly launched experiments for different tasks, seeking model improvements by iteratively fine-tuning them and regularly comparing results against established benchmarks. The data involved in these experiments was diverse, including maps, LiDAR, camera, radar, inertial, and other sensor data. Keeping track of the data collected from these experiments and exporting it in an organized and shareable way became a challenge. The company also identified a lack of tooling for planning and building consistent benchmark datasets. They needed a solution that would allow them to share benchmark results in a constant place and format and retain data for later comparison after the end of a project.

Continuum Industries, a company in the infrastructure industry, developed a product called Optioneer to automate and optimize the design of linear infrastructure assets. However, operating the Optioneer engine presented several challenges. The engine needed to be robust enough to handle different problems fed by different customers. Moreover, the company needed to ensure that the solutions provided by the engine were accurate and agreed upon by civil engineers. The team also had to constantly improve the optimization engine without breaking the algorithm. The nature of the problem they were trying to solve presented additional challenges. They could not automatically tell whether an algorithm output was correct or not. They needed a set of example problems that was representative of the kind of problem that the algorithm would be asked to solve in production. The team initially developed a custom solution to these problems, but it proved to be extremely clunky and complex to maintain.

InstaDeep's BioAI team was faced with the challenge of managing numerous experiments for their DeepChain™ platform, a complex software for protein design. The team was dealing with scattered experiment logs, difficulty in sharing experiment results, and the burden of handling infrastructure and operations. The team needed a solution that was easy to use, could connect to TensorFlow and PyTorch logs, and was cost-effective. The challenge was to find a tool that could streamline their experiment management process, improve visibility of experiment logs, and facilitate collaboration among researchers and engineers.

ailslab, a small bioinformatics research group, is dedicated to building machine learning models to predict cardiovascular disease development using clinical, imaging, and genetics data. The research process is intense, requiring a custom infrastructure to extract features from various data types, including Electronic Health Records (EHR), time-to-event data, images, structured data, and ECG. The goal is to create precise machine learning models for risk stratification for primary cardiovascular prevention. However, as the team grew, collaboration became more challenging, and new problems began to emerge. These included issues with data privacy, workflow standardization, feature and model selection, experiment management, and information logging.

Hypefactors, a technology company specializing in media intelligence and reputation tracking, faced a significant challenge in managing their data pipelines. These pipelines monitor a wide range of media, including social media, print, television, and radio, to analyze changes in their customers' brand reputation. The process involves gathering data from various sources and enriching it with machine learning (ML) features. However, as the company expanded its operations and started working on more complex ML problems, they encountered difficulties in tracking their experiments. Initially, the team used Slack for collaboration and personal notes/files for storing training metadata and model artifacts. However, as the number of models, features, and team members increased, this method became inefficient and created structural bottlenecks.

ReSpo.Vision, a company specializing in sports data analysis, faced significant challenges in managing their machine learning (ML) pipelines at scale. The company uses computer vision and machine learning to extract 3D data from single-view camera sports broadcast videos, providing valuable insights to players, scouts, managers, clubs, and federations. Their ML team collects raw data, labels it, and adds new datasets to training and evaluation pipelines. However, as they scaled up the number of matches processed, the number of pipelines ran to build different models also increased, making it extremely hard to manage the workflow and debug pipeline failures. They had difficulty in debugging issues with the experiment results, figuring out if their pipelines all finished successfully, and comparing the results of each run to the previous runs. Additionally, it was problematic to know what dataset or parameters were used for each experiment run. They needed a better way to manage their pipeline runs and make the best use of their resources.

TH Köln, Germany’s largest University of Applied Sciences in the Electrical Engineering Department, was facing significant challenges in managing large-scale research projects. The department, which focuses on meta-learning research with standard ML frameworks such as TensorFlow and PyTorch, was struggling with experiment-tracking across multiple servers. The team was manually creating CSV files to record details generated during the experiment run such as loss or f2 score. Additionally, there were separate files with hyperparameters and other configurations. This manual management of multiple files made the analysis of past experiments extremely challenging and prone to errors. The team was also facing issues with multi-server project management, access control management, result comparison and presentation, and loss of experiment history when students left.

Zoined, a company offering cloud-based Retail and Hospitality Analytics, faced a significant challenge in tracking and managing experiments, especially with a small team of scientists and engineers. The company's data scientist, Kha, was solely responsible for the forecasting pipeline, making experiment tracking a tedious manual task. Kha was dealing with large data frames with forecasts that needed to be logged alongside their experiments. He also needed a way to visualize results for complete and intermediate experiments to enhance efficiency. The team initially tried using Splunk for experiment tracking, but it proved to be intimidating, difficult for visualizing logged values, and expensive. The next solution, MLflow, presented issues with hosting options, was compute-intensive, and had problems with auto scaling. It also made collaboration difficult as sharing experiments was not straightforward.

The team at Elevatus faced significant challenges in managing their machine learning models due to the lack of an observability layer. This absence led to inefficiencies, such as the need to reanalyze data and reimplement workflows whenever evidence for language models was requested. The team recognized the necessity of a mature observability layer to facilitate large-scale training jobs and involve stakeholders in auditing AI systems. They sought a model tracking solution that could provide historical performance tracking and integrate well with their existing tools, such as PyTorch Lightning and Optuna. The goal was to enhance model performance tracking and optimize compute utilization, enabling quick iteration across experiments.