# Operationalizing Machine Learning: An Interview Study

arXiv: https://arxiv.org/abs/2209.09125 · HTML: https://mlsanity.com/paper/2209.09125/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 79 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 24 of 24)

- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [A Fine-Grained Analysis on Distribution Shift](https://mlsanity.com/paper/2110.11328/) (2021)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [Machine Learning Operations (MLOps): Overview, Definition, and Architecture](https://mlsanity.com/paper/2205.02302/) (2022)
- [Rethinking Importance Weighting for Deep Learning under Distribution Shift](https://mlsanity.com/paper/2006.04662/) (2020)
- [The CLEAR Benchmark: Continual LEArning on Real-World Imagery](https://mlsanity.com/paper/2201.06289/) (2022)
- [Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI](https://mlsanity.com/paper/1909.02309/) (2019)
- [How do Data Science Workers Collaborate? Roles, Workflows, and Tools](https://mlsanity.com/paper/2001.06684/) (2020)
- [Who Needs MLOps: What Data Scientists Seek to Accomplish and How Can MLOps Help?](https://mlsanity.com/paper/2103.08942/) (2021)
- [A Data Quality-Driven View of MLOps](https://mlsanity.com/paper/2102.07750/) (2021)
- [Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities](https://mlsanity.com/paper/2103.16007/) (2021)
- [Whither AutoML? Understanding the Role of Automation in Machine Learning Workflows](https://mlsanity.com/paper/2101.04834/) (2021)
- [A Survey of DevOps Concepts and Challenges](https://mlsanity.com/paper/1909.05409/) (2019)
- [Towards Observability for Production Machine Learning Pipelines](https://mlsanity.com/paper/2108.13557/) (2021)
- [On Continuous Integration / Continuous Delivery for Automated Deployment of Machine Learning Models using MLOps](https://mlsanity.com/paper/2202.03541/) (2022)
- [Rethinking Streaming Machine Learning Evaluation](https://mlsanity.com/paper/2205.11473/) (2022)
- [Designing Tools for Semi-Automated Detection of Machine Learning Biases: An Interview Study](https://mlsanity.com/paper/2003.07680/) (2020)
- [Demystifying a Dark Art: Understanding Real-World Machine Learning Model Development](https://mlsanity.com/paper/2005.01520/) (2020)
- [Fine-Grained Lineage for Safer Notebook Interactions](https://mlsanity.com/paper/2012.06981/) (2020)
- [Hindsight Logging for Model Training](https://mlsanity.com/paper/2006.07357/) (2020)

## Built on by (most cited first, 25 of 25)

- [Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes](https://mlsanity.com/paper/2304.09433/) (2023)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines](https://mlsanity.com/paper/2401.03038/) (2024)
- [Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML](https://mlsanity.com/paper/2303.08485/) (2023)
- [The Data Addition Dilemma](https://mlsanity.com/paper/2408.04154/) (2024)
- [Revisiting Prompt Engineering via Declarative Crowdsourcing](https://mlsanity.com/paper/2308.03854/) (2023)
- [Better Training Data Attribution via Better Inverse Hessian-Vector Products](https://mlsanity.com/paper/2507.14740/) (2025)
- [A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities](https://mlsanity.com/paper/2210.08906/) (2022)
- [Monitoring and Adapting ML Models on Mobile Devices](https://mlsanity.com/paper/2305.07772/) (2023)
- [Beyond Model Interpretability: Socio-Structural Explanations in Machine Learning](https://mlsanity.com/paper/2409.03632/) (2024)
- [Reasonable Scale Machine Learning with Open-Source Metaflow](https://mlsanity.com/paper/2303.11761/) (2023)
- [Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie](https://mlsanity.com/paper/2404.13682/) (2024)
- [JoinBoost: Grow Trees Over Normalized Data Using Only SQL](https://mlsanity.com/paper/2307.00422/) (2023)
- [How to Sustainably Monitor ML-Enabled Systems? Accuracy and Energy Efficiency Tradeoffs in Concept Drift Detection](https://mlsanity.com/paper/2404.19452/) (2024)
- [Measuring Distributional Shifts in Text: The Advantage of Language Model-Based Embeddings](https://mlsanity.com/paper/2312.02337/) (2023)
- [Bauplan: zero-copy, scale-up FaaS for data pipelines](https://mlsanity.com/paper/2410.17465/) (2024)
- [A Metrics-Oriented Architectural Model to Characterize Complexity on Machine Learning-Enabled Systems](https://mlsanity.com/paper/2506.08153/) (2025)
- [Causal Inference out of Control: Estimating the Steerability of Consumption](https://mlsanity.com/paper/2302.04989/) (2023)
- [An Efficient Framework for Monitoring Subgroup Performance of Machine Learning Systems](https://mlsanity.com/paper/2212.08312/) (2022)
- [Quality Issues in Machine Learning Software Systems](https://mlsanity.com/paper/2306.15007/) (2023)
- [Expert-Driven Monitoring of Operational ML Models](https://mlsanity.com/paper/2401.11993/) (2024)
- [On a Scale-Invariant Approach to Bundle Recommendations in Candy Crush Saga](https://mlsanity.com/paper/2408.06799/) (2024)
- [A Tale of Two Systems: Characterizing Architectural Complexity on Machine Learning-Enabled Systems](https://mlsanity.com/paper/2506.11295/) (2025)
- [GPUAlert: A Zero-Instrumentation Process-Boundary Monitor for Diagnosing GPU Training-Job Failures](https://mlsanity.com/paper/2607.01409/) (2026)
- [Agentic Self-Healing for Data and AI Pipelines: An Affordable Vendor-Agnostic Architecture using Open-Source Software](https://mlsanity.com/paper/2608.01955/) (2026)

## Abstract (arXiv)

Organizations rely on machine learning engineers (MLEs) to operationalize ML, i.e., deploy and maintain ML pipelines in production. The process of operationalizing ML, or MLOps, consists of a continual loop of (i) data collection and labeling, (ii) experimentation to improve ML performance, (iii) evaluation throughout a multi-staged deployment process, and (iv) monitoring of performance drops in production. When considered together, these responsibilities seem staggering -- how does anyone do MLOps, what are the unaddressed challenges, and what are the implications for tool builders? We conducted semi-structured ethnographic interviews with 18 MLEs working across many applications, including chatbots, autonomous vehicles, and finance. Our interviews expose three variables that govern success for a production ML deployment: Velocity, Validation, and Versioning. We summarize common practices for successful ML experimentation, deployment, and sustaining production performance. Finally, we discuss interviewees' pain points and anti-patterns, with implications for tool design.