mlsanityData to 14 Oct 2026

Learning to Select Visual In-Context Demonstrations

arXiv 2603.26775 ↗First posted 24 Mar 2026cs.LG cs.AI cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) adapt to visual tasks via in-context learning (ICL), which relies heavily on demonstration quality. The dominant demonstration selection strategy is unsupervised k-Nearest Neighbor (kNN) search. While simple, this similarity-first approach is sub-optimal for complex factual regression tasks; it selects redundant examples that fail to capture the task's full output range. We reframe selection as a sequential decision-making problem and introduce Learning to Select Demonstrations (LSD), training a Reinforcement Learning agent to construct optimal demonstration sets. Using a Dueling DQN with a query-centric Transformer Decoder, our agent learns a policy that maximizes MLLM downstream performance. Evaluating across five visual regression benchmarks, we uncover a crucial dichotomy: while kNN remains optimal for subjective preference tasks, LSD significantly outperforms baselines on objective, factual regression tasks. By balancing visual relevance with diversity, LSD better defines regression boundaries, illuminating when learned selection is strictly necessary for visual ICL.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 60 references that are in the map. First posted on arXiv 24 Mar 2026.

Builds on

46 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations5 Mar

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2025Gemma 3 Technical Reportcited by 1,824
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  7. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  8. 2024Phi-4 Technical Reportcited by 711
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  11. 2021Editing Factual Knowledge in Language Modelscited by 553
  12. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
33 more
  1. 2019KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessmentcited by 355
  2. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  3. 2022Is GPT-3 a Good Data Annotator?cited by 208
  4. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  5. 2022Active Example Selection for In-Context Learningcited by 196
  6. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
  7. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  8. 2023Compositional Exemplars for In-context Learningcited by 144
  9. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 128
  10. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  11. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  12. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  13. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generatorcited by 68
  14. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 54
  15. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  16. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  17. 2023Using In-Context Learning to Improve Dialogue Safetycited by 42
  18. 2018SCUT-FBP5500: A Diverse Benchmark Dataset for Multi-Paradigm Facial Beauty Predictioncited by 33
  19. 2023Exploring In-Context Learning Capabilities of Foundation Models for Generating Knowledge Graphs from Textcited by 19
  20. 2023A Call to Reflect on Evaluation Practices for Age Estimation: Comparative Analysis of the State-of-the-Art and a Unified Benchmarkcited by 19
  21. 2023Auto-ICL: In-Context Learning without Human Supervisioncited by 18
  22. 2025Sample Efficient Demonstration Selection for In-Context Learningcited by 13
  23. 2019Deep Instance-Level Hard Negative Mining Model for Histopathology Imagescited by 12
  24. 2024Large Language Models are Contrastive Reasonerscited by 11
  25. 2024Demonstration Selection for In-Context Learning via Reinforcement Learningcited by 8
  26. 2025The Role of Diversity in In-Context Learning for Large Language Modelscited by 6
  27. 2018Transferring Rich Deep Features for Facial Beauty Predictioncited by 5
  28. 2025Learning to Select In-Context Demonstration Preferred by Large Language Modelcited by 5
  29. 2021Regression or Classification? New Methods to Evaluate No-Reference Picture and Video Quality Modelscited by 4
  30. 2024Improving In-Context Learning with Prediction Feedback for Sentiment Analysiscited by 4
  31. 2025Exploring Task-Level Optimal Prompts for Visual In-Context Learningcited by 4
  32. 2025Towards Reliable and Holistic Visual In-Context Learning Prompt Selectioncited by 2
  33. 2024What Makes Good Few-shot Examples for Vision-Language Models?cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026In-Context Learning for Wound Classification with Small Multimodal Language Modelscited by 0
  2. 2026ActICL: Post-Training for Agentic Multimodal In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning
  2. 2025Efficient and Effective In-context Demonstration Selection with Coreset
  3. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
  4. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization
  5. 2026In-Context Learning Can Help Vision-Language Models Overcome Training Priors
  6. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  7. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning
  8. 2025Leveraging In-Context Learning for Language Model Agents