# Agile Modeling: From Concept to Classifier in Minutes

arXiv: https://arxiv.org/abs/2302.12948 · HTML: https://mlsanity.com/paper/2302.12948/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 81 references that are in the map.
- First posted on arXiv 25 Feb 2023.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples](https://mlsanity.com/paper/1903.03096/) (2019)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Batch Active Learning at Scale](https://mlsanity.com/paper/2107.14263/) (2021)
- [Deep Active Learning for Named Entity Recognition](https://mlsanity.com/paper/1707.05928/) (2017)
- ["This is my unicorn, Fluffy": Personalizing frozen vision-language representations](https://mlsanity.com/paper/2204.01694/) (2022)
- [Bayesian Batch Active Learning as Sparse Subset Approximation](https://mlsanity.com/paper/1908.02144/) (2019)
- [Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering](https://mlsanity.com/paper/2107.02331/) (2021)
- [Quantization based Fast Inner Product Search](https://mlsanity.com/paper/1509.01469/) (2015)
- [Similarity Search for Efficient Active Learning and Search of Rare Concepts](https://mlsanity.com/paper/2007.00077/) (2020)
- [Inferring Generative Model Structure with Static Analysis](https://mlsanity.com/paper/1709.02477/) (2017)
- [EQUI-VOCAL: Synthesizing Queries for Compositional Video Events from Limited User Interactions [Technical Report]](https://mlsanity.com/paper/2301.00929/) (2023)
- [Background Splitting: Finding Rare Classes in a Sea of Background](https://mlsanity.com/paper/2008.12873/) (2020)

## Built on by (most cited first, 8 of 8)

- [Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models](https://mlsanity.com/paper/2312.03052/) (2023)
- [m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks](https://mlsanity.com/paper/2403.11085/) (2024)
- [Designing LLM Chains by Adapting Techniques from Crowdsourcing Workflows](https://mlsanity.com/paper/2312.11681/) (2023)
- [Semantic and Expressive Variation in Image Captions Across Languages](https://mlsanity.com/paper/2310.14356/) (2023)
- [Leveraging tropical reef, bird and unrelated sounds for superior transfer learning in marine bioacoustics](https://mlsanity.com/paper/2404.16436/) (2024)
- [From Categories to Classifiers: Name-Only Continual Learning by Exploring the Web](https://mlsanity.com/paper/2311.11293/) (2023)
- [A Zero-Shot Learning Approach for Ephemeral Gully Detection from Remote Sensing using Vision Language Models](https://mlsanity.com/paper/2503.01169/) (2025)
- [Agile Deliberation: Concept Deliberation for Subjective Visual Classification](https://mlsanity.com/paper/2512.10821/) (2025)

## Abstract (arXiv)

The application of computer vision to nuanced subjective use cases is growing. While crowdsourcing has served the vision community well for most objective tasks (such as labeling a "zebra"), it now falters on tasks where there is substantial subjectivity in the concept (such as identifying "gourmet tuna"). However, empowering any user to develop a classifier for their concept is technically difficult: users are neither machine learning experts, nor have the patience to label thousands of examples. In reaction, we introduce the problem of Agile Modeling: the process of turning any subjective visual concept into a computer vision model through a real-time user-in-the-loop interactions. We instantiate an Agile Modeling prototype for image classification and show through a user study (N=14) that users can create classifiers with minimal effort under 30 minutes. We compare this user driven process with the traditional crowdsourcing paradigm and find that the crowd's notion often differs from that of the user's, especially as the concepts become more subjective. Finally, we scale our experiments with simulations of users training classifiers for ImageNet21k categories to further demonstrate the efficacy.