mlsanityData to 7 Oct 2026

VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning

arXiv 2403.13164 ↗First posted 19 Mar 2024cs.LGcited by 34

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) famously exhibit emergent in-context learning (ICL) -- the ability to rapidly adapt to new tasks using few-shot examples provided as a prompt, without updating the model's weights. Built on top of LLMs, vision large language models (VLLMs) have advanced significantly in areas such as recognition, reasoning, and grounding. However, investigations into multimodal ICL have predominantly focused on few-shot visual question answering (VQA), and image captioning, which we will show neither exploit the strengths of ICL, nor test its limitations. The broader capabilities and limitations of multimodal ICL remain under-explored. In this study, we introduce a comprehensive benchmark VL-ICL Bench for multimodal in-context learning, encompassing a broad spectrum of tasks that involve both images and text as inputs and outputs, and different types of challenges, from perception to reasoning and long context length. We evaluate the abilities of state-of-the-art VLLMs against this benchmark suite, revealing their diverse strengths and weaknesses, and showing that even the most advanced models, such as GPT-4, find the tasks challenging. By highlighting a range of new ICL tasks, and the associated strengths and limitations of existing models, we hope that our dataset will inspire future work on enhancing the in-context learning capabilities of VLLMs, as well as inspire new applications that leverage VLLM ICL. The code and dataset are available at https://github.com/ys-zong/VL-ICL.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 49 of its 69 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

49 of 69 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
37 more
  1. 2016Matching Networks for One Shot Learningcited by 2,246
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  13. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  14. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  15. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  17. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  18. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  19. 2021MetaICL: Learning to Learn In Contextcited by 450
  20. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  21. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  22. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  23. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  24. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  25. 2023Generating Images with Multimodal Language Modelscited by 259
  26. 2022Visual Prompting via Image Inpaintingcited by 248
  27. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  28. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  29. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  30. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  31. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  32. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  33. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  34. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  35. 2022T-NER: An All-Round Python Library for Transformer-based Named Entity Recognitioncited by 37
  36. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 31
  37. 2024Can MLLMs Perform Text-to-Image In-Context Learning?cited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 20.

34 papers.

Sort:

By area:

  1. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  2. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configurationcited by 23
  3. 2024LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generationcited by 19
  4. 2025CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learningcited by 18
  5. 2026QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understandingcited by 9
  6. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrationscited by 5
  7. 2025SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learningcited by 5
  8. 2025Generalizable Object Re-Identification via Visual In-Context Promptingcited by 4
  9. 2025CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?cited by 3
  10. 2025PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Modelscited by 3
  11. 2025Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoningcited by 2
  12. 2026EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learningcited by 2
22 more
  1. 2026UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomycited by 2
  2. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 2
  3. 2024Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Modelscited by 1
  4. 2026ArtifactLens: Hundreds of Labels Are Enough for Artifact Detection with VLMscited by 1
  5. 2026MMCL-Bench: Multimodal Context Learning from Visual Rules, Procedures, and Evidencecited by 1
  6. 2024ANCHOR: LLM-driven Subject Conditioning for Text-to-Image Synthesiscited by 0
  7. 2025Image-Text Relation Prediction for Multilingual Tweetscited by 0
  8. 2025Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answeringcited by 0
  9. 2026FewMMBench: A Benchmark for Multimodal Few-Shot Learningcited by 0
  10. 2026Parallel In-context Learning for Large Vision Language Modelscited by 0
  11. 2026Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generationcited by 0
  12. 2026Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Taskscited by 0
  13. 2026From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Modelscited by 0
  14. 2026Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Modelscited by 0
  15. 2026In-Context Collapse in Vision-Language Models and How to Mitigate it?cited by 0
  16. 2026MAG: MAnifold Guided Semi-Supervised Multi-modal In-Context Learningcited by 0
  17. 2026Taming Visual Neglect: A Variational Information Bottleneck Framework for Adaptive Attention in Multimodal In-Context Learningcited by 0
  18. 2026Instruction Distillation: Text Instructions as Visual Examplescited by 0
  19. 2026TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactualscited by 0
  20. 2026V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environmentscited by 0
  21. 2026Are In-Context Images Worth 10 Dimensions?cited by 0
  22. 2026Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Modelscited by 0