mlsanityData to 7 Oct 2026

Visual In-Context Learning for Large Vision-Language Models

arXiv 2402.11574 ↗First posted 18 Feb 2024cs.CV cs.CLcited by 124

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

In Large Visual Language Models (LVLMs), the efficacy of In-Context Learning (ICL) remains limited by challenges in cross-modal interactions and representation disparities. To overcome these challenges, we introduce a novel Visual In-Context Learning (VICL) method comprising Visual Demonstration Retrieval, Intent-Oriented Image Summarization, and Intent-Oriented Demonstration Composition. Our approach retrieves images via ''Retrieval & Rerank'' paradigm, summarises images with task intent and task-specific visual parsing, and composes language-based demonstrations that reduce token count and alleviate cross-modal interaction problem. Experimental evaluations on five visual reasoning datasets demonstrate the effectiveness of our method. Moreover, our extensive experiments leverage information flow analysis to elucidate the effectiveness of our method, and investigate the impact of length and position of demonstrations for LVLM. The use of in-context unlearning further shows promise in resetting specific model knowledge without retraining.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 51 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

36 of 51 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2019Machine Unlearningcited by 1,020
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2023A Survey on Multimodal Large Language Modelscited by 898
  12. 2019Are Sixteen Heads Really Better than One?cited by 803
24 more
  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  3. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  4. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  5. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  6. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  7. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  8. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  9. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  10. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  11. 2022A very preliminary analysis of DALL-E 2cited by 82
  12. 2023EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributescited by 81
  13. 2023Few-shot In-context Learning for Knowledge Base Question Answeringcited by 74
  14. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  15. 2023Thread of Thought Unraveling Chaotic Contextscited by 71
  16. 2023Adaptive Machine Translation with Large Language Modelscited by 58
  17. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  18. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  19. 2018Contemplating Visual Emotions: Understanding and Overcoming Dataset Biascited by 41
  20. 2022Towards Robust Ranker for Text Retrievalcited by 37
  21. 2023Finding and Editing Multi-Modal Neurons in Pre-Trained Transformerscited by 36
  22. 2023Understanding Emergent In-Context Learning from a Kernel Regression Perspectivecited by 23
  23. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  24. 2023IMProv: Inpainting-based Multimodal Prompting for Computer Vision Taskscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 34; 2025: 65; 2026: 25.

124 papers.

Sort:

By area:

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
  4. 2025OmniSVG: A Unified Scalable Vector Graphics Generation Modelcited by 71
  5. 2025VisualCloze: A Universal Image Generation Framework via Visual In-Context Learningcited by 39
  6. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  7. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  8. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  9. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  10. 2024Residual-based Language Models are Free Boosters for Biomedical Imagingcited by 23
  11. 2024Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detectioncited by 22
  12. 2024Automatic News Generation and Fact-Checking System Based on Language Processingcited by 22
38 more
  1. 2025ComplexBench-Edit: Benchmarking Complex Instruction-Driven Image Editing via Compositional Dependenciescited by 20
  2. 2025Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counselingcited by 17
  3. 2024Towards Global Optimal Visual In-Context Learning Prompt Selectioncited by 15
  4. 2024Rumor Detection with a novel graph neural network approachcited by 14
  5. 2024The New Agronomists: Language Models are Experts in Crop Managementcited by 12
  6. 2024Image Captioning in news report scenariocited by 11
  7. 2024InsectMamba: Insect Pest Classification with State Space Modelcited by 11
  8. 20253D Question Answering via only 2D Vision-Language Modelscited by 10
  9. 2024Research on the Application of Computer Vision Based on Deep Learning in Autonomous Driving Technologycited by 9
  10. 2025Semantic Causality-Aware Vision-Based 3D Occupancy Predictioncited by 8
  11. 2026Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisitioncited by 8
  12. 2024Mapping New Realities: Ground Truth Image Creation with Pix2Pix Image-to-Image Translationcited by 7
  13. 2025Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learningcited by 7
  14. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  15. 2025An LLM-LVLM Driven Agent for Iterative and Fine-Grained Image Editingcited by 6
  16. 2024Educational Personalized Learning Path Planning with Large Language Modelscited by 5
  17. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrationscited by 5
  18. 2025When and How Unlabeled Data Provably Improve In-Context Learningcited by 5
  19. 2025Contextualizing Search Queries In-Context Learning for Conversational Rewriting with LLMscited by 4
  20. 2025ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Modelscited by 4
  21. 2025X-Ray-CoT: Interpretable Chest X-ray Diagnosis with Vision-Language Models via Chain-of-Thought Reasoningcited by 4
  22. 2024Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?cited by 3
  23. 2025End-to-End Chart Summarization via Visual Chain-of-Thought in Vision-Language Modelscited by 3
  24. 2025Interpretable and Robust Dialogue State Tracking via Natural Language Summarization with LLMscited by 3
  25. 2025VCM: Vision Concept Modeling Based on Implicit Contrastive Learning with Vision-Language Instruction Fine-Tuningcited by 3
  26. 2025ByDeWay: Boost Your multimodal LLM with DEpth prompting in a Training-Free Waycited by 3
  27. 2025Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMscited by 3
  28. 2025ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Followingcited by 3
  29. 2026Retrieving Counterfactuals Improves Visual In-Context Learningcited by 3
  30. 2026Demystifying Low-Rank Knowledge Distillation in Large Language Models: Convergence, Generalization, and Information-Theoretic Guaranteescited by 3
  31. 2024Optimizing Psychological Counseling with Instruction-Tuned Large Language Modelscited by 2
  32. 2024Improving Visual Storytelling with Multimodal Large Language Modelscited by 2
  33. 2024Large Visual-Language Models Are Also Good Classifiers: A Study of In-Context Multimodal Fake News Detectioncited by 2
  34. 2024Can LLMs predict the convergence of Stochastic Gradient Descent?cited by 2
  35. 2024Large Vision-Language Models for Remote Sensing Visual Question Answeringcited by 2
  36. 2025Contrastive Speaker-Aware Learning for Multi-party Dialogue Generation with LLMscited by 2
  37. 2025High-Fidelity Pseudo-label Generation by Large Language Models for Training Robust Radiology Report Classifierscited by 2
  38. 2025Knowledge-Augmented Language Models Interpreting Structured Chest X-Ray Findingscited by 2

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  2. 2025T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs
  3. 2023Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties
  4. 2023MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models
  5. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  6. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  7. 2026In-Context Collapse in Vision-Language Models and How to Mitigate it?
  8. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning