# Mind the (Data) Gap: Evaluating Vision Systems in Small Data Applications

arXiv: https://arxiv.org/abs/2504.06486 · HTML: https://mlsanity.com/paper/2504.06486/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 44 references that are in the map.
- First posted on arXiv 8 Apr 2025.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [Benchmarking Representation Learning for Natural World Image Collections](https://mlsanity.com/paper/2103.16483/) (2021)
- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [Multimodal Autoregressive Pre-training of Large Vision Encoders](https://mlsanity.com/paper/2411.14402/) (2024)
- [EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters](https://mlsanity.com/paper/2402.04252/) (2024)
- [The iWildCam 2021 Competition Dataset](https://mlsanity.com/paper/2105.03494/) (2021)
- [Inverse scaling can become U-shaped](https://mlsanity.com/paper/2211.02011/) (2022)
- [Many-Shot In-Context Learning in Multimodal Foundation Models](https://mlsanity.com/paper/2405.09798/) (2024)

## Built on by (most cited first, 3 of 3)

- [BeetleVerse: A Study on Taxonomic Classification of Ground Beetles](https://mlsanity.com/paper/2504.13393/) (2025)
- [Visual Species Recognition with Large Multimodal Models as Post-Hoc Correctors](https://mlsanity.com/paper/2512.15748/) (2025)
- [A continental-scale dataset of ground beetles with high-resolution images and validated morphological trait measurements](https://mlsanity.com/paper/2601.10687/) (2026)

## Abstract (arXiv)

The practical application of AI tools for specific computer vision tasks relies on the "small-data regime" of hundreds to thousands of labeled samples. This small-data regime is vital for applications requiring expensive expert annotations, such as ecological monitoring, medical diagnostics or industrial quality control. We find, however, that computer vision research has ignored the small data regime as evaluations increasingly focus on zero- and few-shot learning. We use the Natural World Tasks (NeWT) benchmark to compare multi-modal large language models (MLLMs) and vision-only methods across varying training set sizes. MLLMs exhibit early performance plateaus, while vision-only methods improve throughout the small-data regime, with performance gaps widening beyond 10 training examples. We provide the first comprehensive comparison between these approaches in small-data contexts and advocate for explicit small-data evaluations in AI research to better bridge theoretical advances with practical deployments.