# How Developers Iterate on Machine Learning Workflows -- A Survey of the Applied Machine Learning Literature

arXiv: https://arxiv.org/abs/1803.10311 · HTML: https://mlsanity.com/paper/1803.10311/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2018.

## Built on by (most cited first, 6 of 6)

- [Position: A Call to Action for a Human-Centered AutoML Paradigm](https://mlsanity.com/paper/2406.03348/) (2024)
- [Globus Automation Services: Research process automation across the space-time continuum](https://mlsanity.com/paper/2208.09513/) (2022)
- [Bauplan: zero-copy, scale-up FaaS for data pipelines](https://mlsanity.com/paper/2410.17465/) (2024)
- [LLM Personas as a Substitute for Field Experiments in Method Benchmarking](https://mlsanity.com/paper/2512.21080/) (2025)
- [Streamlining Admission with LOR Insights: AI-Based Leadership Assessment in Online Master's Program](https://mlsanity.com/paper/2508.05513/) (2025)
- [ML in a Box: Analyzing Containerization Practices in Open Source ML Projects](https://mlsanity.com/paper/2607.10126/) (2026)

## Abstract (arXiv)

Machine learning workflow development is anecdotally regarded to be an iterative process of trial-and-error with humans-in-the-loop. However, we are not aware of quantitative evidence corroborating this popular belief. A quantitative characterization of iteration can serve as a benchmark for machine learning workflow development in practice, and can aid the development of human-in-the-loop machine learning systems. To this end, we conduct a small-scale survey of the applied machine learning literature from five distinct application domains. We collect and distill statistics on the role of iteration within machine learning workflow development, and report preliminary trends and insights from our investigation, as a starting point towards this benchmark. Based on our findings, we finally describe desiderata for effective and versatile human-in-the-loop machine learning systems that can cater to users in diverse domains.