# Accelerating Human-in-the-loop Machine Learning: Challenges and Opportunities

arXiv: https://arxiv.org/abs/1804.05892 · HTML: https://mlsanity.com/paper/1804.05892/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2018.

## Built on by (most cited first, 20 of 20)

- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [High-Definition Map Generation Technologies For Autonomous Driving](https://mlsanity.com/paper/2206.05400/) (2022)
- [Computing Multiple Image Reconstructions with a Single Hypernetwork](https://mlsanity.com/paper/2202.11009/) (2022)
- [Evaluation of Induced Expert Knowledge in Causal Structure Learning by NOTEARS](https://mlsanity.com/paper/2301.01817/) (2023)
- [Dual-AEB: Synergizing Rule-Based and Multimodal Large Language Models for Effective Emergency Braking](https://mlsanity.com/paper/2410.08616/) (2024)
- [Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach](https://mlsanity.com/paper/2410.21346/) (2024)
- [Stochastic Trajectory Prediction under Unstructured Constraints](https://mlsanity.com/paper/2503.14203/) (2025)
- [VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR](https://mlsanity.com/paper/2505.17423/) (2025)
- [Active learning for medical code assignment](https://mlsanity.com/paper/2104.05741/) (2021)
- [Smart Language Agents in Real-World Planning](https://mlsanity.com/paper/2407.19667/) (2024)
- [HP-BERT: A framework for longitudinal study of Hinduphobia on social media via language models](https://mlsanity.com/paper/2501.05482/) (2025)
- [Interactive Analysis of Global Explanations using Aggregated Class Activation Maps for Network Data](https://mlsanity.com/paper/2608.13575/) (2026)
- [Towards Model-informed Precision Dosing with Expert-in-the-loop Machine Learning](https://mlsanity.com/paper/2106.14384/) (2021)
- [Power of human-algorithm collaboration in solving combinatorial optimization problems](https://mlsanity.com/paper/2107.11784/) (2021)
- [A Blockchain Protocol for Human-in-the-Loop AI](https://mlsanity.com/paper/2211.10859/) (2022)
- [Transforming Unstructured Text into Data with Context Rule Assisted Machine Learning (CRAML)](https://mlsanity.com/paper/2301.08549/) (2023)
- [MaskSearch: Querying Image Masks at Scale](https://mlsanity.com/paper/2305.02375/) (2023)
- [Efficient Failure Pattern Identification of Predictive Algorithms](https://mlsanity.com/paper/2306.00760/) (2023)
- [Assurance for Deployed Continual Learning Systems](https://mlsanity.com/paper/2311.10787/) (2023)
- [The Machine Learning Canvas: Empirical Findings on Why Strategy Matters More Than AI Code Generation](https://mlsanity.com/paper/2601.01839/) (2026)

## Abstract (arXiv)

Development of machine learning (ML) workflows is a tedious process of iterative experimentation: developers repeatedly make changes to workflows until the desired accuracy is attained. We describe our vision for a "human-in-the-loop" ML system that accelerates this process: by intelligently tracking changes and intermediate results over time, such a system can enable rapid iteration, quick responsive feedback, introspection and debugging, and background execution and automation. We finally describe Helix, our preliminary attempt at such a system that has already led to speedups of up to 10x on typical iterative workflows against competing systems.