# ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems

arXiv: https://arxiv.org/abs/1409.4814 · HTML: https://mlsanity.com/paper/1409.4814/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2014.

## Built on by (most cited first, 7 of 7)

- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [Putting Humans in the Natural Language Processing Loop: A Survey](https://mlsanity.com/paper/2103.04044/) (2021)
- [ScatterShot: Interactive In-context Example Curation for Text Transformation](https://mlsanity.com/paper/2302.07346/) (2023)
- [How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond](https://mlsanity.com/paper/2501.05714/) (2025)
- [A Base Camp for Scaling AI](https://mlsanity.com/paper/1612.07896/) (2016)
- [Human-AI Interaction Design in Machine Teaching](https://mlsanity.com/paper/2206.05182/) (2022)
- [Topic-time Heatmaps for Human-in-the-loop Topic Detection and Tracking](https://mlsanity.com/paper/2110.07337/) (2021)

## Abstract (arXiv)

Quick interaction between a human teacher and a learning machine presents numerous benefits and challenges when working with web-scale data. The human teacher guides the machine towards accomplishing the task of interest. The learning machine leverages big data to find examples that maximize the training value of its interaction with the teacher. When the teacher is restricted to labeling examples selected by the machine, this problem is an instance of active learning. When the teacher can provide additional information to the machine (e.g., suggestions on what examples or predictive features should be used) as the learning task progresses, then the problem becomes one of interactive learning. To accommodate the two-way communication channel needed for efficient interactive learning, the teacher and the machine need an environment that supports an interaction language. The machine can access, process, and summarize more examples than the teacher can see in a lifetime. Based on the machine's output, the teacher can revise the definition of the task or make it more precise. Both the teacher and the machine continuously learn and benefit from the interaction. We have built a platform to (1) produce valuable and deployable models and (2) support research on both the machine learning and user interface challenges of the interactive learning problem. The platform relies on a dedicated, low-latency, distributed, in-memory architecture that allows us to construct web-scale learning machines with quick interaction speed. The purpose of this paper is to describe this architecture and demonstrate how it supports our research efforts. Preliminary results are presented as illustrations of the architecture but are not the primary focus of the paper.