# Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities

arXiv: https://arxiv.org/abs/2103.16007 · HTML: https://mlsanity.com/paper/2103.16007/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 53 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 7 of 7)

- [How do Data Science Workers Collaborate? Roles, Workflows, and Tools](https://mlsanity.com/paper/2001.06684/) (2020)
- [Extending Relational Query Processing with ML Inference](https://mlsanity.com/paper/1911.00231/) (2019)
- [Helix: Holistic Optimization for Accelerating Iterative Machine Learning](https://mlsanity.com/paper/1812.05762/) (2018)
- [The Relational Data Borg is Learning](https://mlsanity.com/paper/2008.07864/) (2020)
- [Demystifying a Dark Art: Understanding Real-World Machine Learning Model Development](https://mlsanity.com/paper/2005.01520/) (2020)
- [Understanding Data Science Lifecycle Provenance via Graph Segmentation and Summarization](https://mlsanity.com/paper/1810.04599/) (2018)
- [Aggregation by Provenance Types: A Technique for Summarising Provenance Graphs](https://mlsanity.com/paper/1504.02616/) (2015)

## Built on by (most cited first, 23 of 23)

- [Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training](https://mlsanity.com/paper/2108.09373/) (2021)
- [Operationalizing Machine Learning: An Interview Study](https://mlsanity.com/paper/2209.09125/) (2022)
- [The Pipeline for the Continuous Development of Artificial Intelligence Models -- Current State of Research and Practice](https://mlsanity.com/paper/2301.09001/) (2023)
- [Challenges and Practices of Deep Learning Model Reengineering: A Case Study on Computer Vision](https://mlsanity.com/paper/2303.07476/) (2023)
- [Bayesian Optimization of Function Networks with Partial Evaluations](https://mlsanity.com/paper/2311.02146/) (2023)
- [Preventing Machine Learning Poisoning Attacks Using Authentication and Provenance](https://mlsanity.com/paper/2105.10051/) (2021)
- [Modyn: Data-Centric Machine Learning Pipeline Orchestration](https://mlsanity.com/paper/2312.06254/) (2023)
- [AnyMatch -- Efficient Zero-Shot Entity Matching with a Small Language Model](https://mlsanity.com/paper/2409.04073/) (2024)
- [An Empirical Study of Challenges in Machine Learning Asset Management](https://mlsanity.com/paper/2402.15990/) (2024)
- [Approximate Nearest Neighbour Search on Dynamic Datasets: An Investigation](https://mlsanity.com/paper/2404.19284/) (2024)
- [Position: Stop Reactively Patching Your Model Every Time and Start Proactive Test-Driven AI Development](https://mlsanity.com/paper/2607.20532/) (2026)
- [TraceFL: Interpretability-Driven Debugging in Federated Learning via Neuron Provenance](https://mlsanity.com/paper/2312.13632/) (2023)
- [AntDT: A Self-Adaptive Distributed Training Framework for Leader and Straggler Nodes](https://mlsanity.com/paper/2404.09679/) (2024)
- [stratum: A System Infrastructure for Massive Agent-Centric ML Workloads](https://mlsanity.com/paper/2603.03589/) (2026)
- [EPOCH: An Agentic Protocol for Multi-Round System Optimization](https://mlsanity.com/paper/2603.09049/) (2026)
- [SemPiper: Interactive Code Synthesis for Semantic Operators in Machine Learning Pipelines](https://mlsanity.com/paper/2606.14361/) (2026)
- [Machine Learning Featurizations for AI Hacking of Political Systems](https://mlsanity.com/paper/2110.09231/) (2021)
- [CMLCompiler: A Unified Compiler for Classical Machine Learning](https://mlsanity.com/paper/2301.13441/) (2023)
- [VeML: An End-to-End Machine Learning Lifecycle for Large-scale and High-dimensional Data](https://mlsanity.com/paper/2304.13037/) (2023)
- [Couler: Unified Machine Learning Workflow Optimization in Cloud](https://mlsanity.com/paper/2403.07608/) (2024)
- [AI Factories: It's time to rethink the Cloud-HPC divide](https://mlsanity.com/paper/2509.12849/) (2025)
- [FlowPipe: LLM-Enhanced Conditional Generative Flow Networks for Data Preparation Pipeline Construction](https://mlsanity.com/paper/2606.24679/) (2026)
- [ML Engineering Agents - A Data Management Perspective](https://mlsanity.com/paper/2610.16919/) (2026)

## Abstract (arXiv)

Machine learning (ML) is now commonplace, powering data-driven applications in various organizations. Unlike the traditional perception of ML in research, ML production pipelines are complex, with many interlocking analytical components beyond training, whose sub-parts are often run multiple times on overlapping subsets of data. However, there is a lack of quantitative evidence regarding the lifespan, architecture, frequency, and complexity of these pipelines to understand how data management research can be used to make them more efficient, effective, robust, and reproducible. To that end, we analyze the provenance graphs of 3000 production ML pipelines at Google, comprising over 450,000 models trained, spanning a period of over four months, in an effort to understand the complexity and challenges underlying production ML. Our analysis reveals the characteristics, components, and topologies of typical industry-strength ML pipelines at various granularities. Along the way, we introduce a specialized data model for representing and reasoning about repeatedly run components in these ML pipelines, which we call model graphlets. We identify several rich opportunities for optimization, leveraging traditional data management ideas. We show how targeting even one of these opportunities, i.e., identifying and pruning wasted computation that does not translate to model deployment, can reduce wasted computation cost by 50% without compromising the model deployment cadence.