mlsanityData to 7 Oct 2026

Online In-Context Distillation for Low-Resource Vision Language Models

arXiv 2510.18117 ↗First posted 20 Oct 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

As the field continues its push for ever more resources, this work turns the spotlight on a critical question: how can vision-language models (VLMs) be adapted to thrive in low-resource, budget-constrained settings? While large VLMs offer strong performance, they are impractical to deploy in such settings. Small VLMs, on the other hand, are efficient but typically require costly fine-tuning to close the performance gap with larger models in the deployment domain. Inspired by the in-context learning framework, we propose an online In-Context Distillation (ICD) method, in which a small VLM collaborates with a stronger teacher model at inference time, distilling its knowledge via sparse demonstrations to efficiently bridge the gap between them. Our method is built on an in-depth analysis that identifies the scale and the choice of models for which vision-language ICL is currently feasible, and demonstrates the advantage of ICL over fine-tuning under constrained compute budgets. We enhance our method with a novel cross-modal demonstration selection strategy, teacher test-time scaling to reduce noise, and student uncertainty conditioning to dynamically populate a demonstration pool and minimize teacher queries. Our ICD method significantly boosts the performance of small models (up to 33%) using scarce teacher annotations (as low as 4%), and competes with the teacher's zero-shot performance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 60 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

49 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Visual In-Context Learning for Large Vision-Language Models18 Feb
  2. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models5 Dec
  3. 2023How to Configure Good In-Context Sequence for Visual Question Answering4 Dec
  4. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2025s1: Simple test-time scalingcited by 1,311
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
32 more
  1. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  2. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  4. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  5. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  7. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  8. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  9. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  10. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  11. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  12. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  13. 2022Is GPT-3 a Good Data Annotator?cited by 208
  14. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  15. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  16. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  17. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  18. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  19. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  20. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  21. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  22. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  23. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  24. 2023Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrationscited by 42
  25. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  26. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 31
  27. 2023Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learningcited by 29
  28. 2024Do LLMs estimate uncertainty well in instruction-following?cited by 26
  29. 2023Explaining Autonomous Driving Actions with Visual Question Answeringcited by 16
  30. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  31. 2022StyleBabel: Artistic Style Tagging and Captioningcited by 12
  32. 2023SINC: Self-Supervised In-Context Learning for Vision-Language Taskscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Ego: Embedding-Guided Personalization of Vision-Language Modelscited by 0
  2. 2026Answer-Conditioned Chain-of-Thought Distillation for Few-Shot Industrial Vision with Small VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  2. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  3. 2026MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models
  4. 2023MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models
  5. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  6. 2026In-Context Collapse in Vision-Language Models and How to Mitigate it?
  7. 2026On-Policy Distillation for Vision-Language Model Adaptation, an Effective Paradigm on Low-Quality Multimodal Data
  8. 2026Parallel In-context Learning for Large Vision Language Models