mlsanityData to 7 Oct 2026

MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models

arXiv 2306.01311 ↗First posted 2 Jun 2023cs.CLcited by 12

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large-scale language models have shown the ability to adapt to a new task via conditioning on a few demonstrations (i.e., in-context learning). However, in the vision-language domain, most large-scale pre-trained vision-language (VL) models do not possess the ability to conduct in-context learning. How can we enable in-context learning for VL models? In this paper, we study an interesting hypothesis: can we transfer the in-context learning ability from the language domain to VL domain? Specifically, we first meta-trains a language model to perform in-context learning on NLP tasks (as in MetaICL); then we transfer this model to perform VL tasks by attaching a visual encoder. Our experiments suggest that indeed in-context learning ability can be transferred cross modalities: our model considerably improves the in-context learning capability on VL tasks and can even compensate for the size of the model significantly. On VQA, OK-VQA, and GQA, our method could outperform the baseline model while having 20 times fewer parameters.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 24 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

17 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
3 more
  1. 2021MetaICL: Learning to Learn In Contextcited by 450
  2. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  3. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 2; 2026: 1.

12 papers.

  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  3. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  4. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  5. 2023Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?cited by 31
  6. 2023Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Propertiescited by 21
  7. 2023Large Multimodal Models: Notes on CVPR 2023 Tutorialcited by 18
  8. 2024Can MLLMs Perform Text-to-Image In-Context Learning?cited by 13
  9. 2024Large Visual-Language Models Are Also Good Classifiers: A Study of In-Context Multimodal Fake News Detectioncited by 2
  10. 2025COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shiftscited by 1
  11. 2025JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Constructioncited by 0
  12. 2026Personalized Object Identification and Localization via In-Context Inference with Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  2. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  3. 2024Visual In-Context Learning for Large Vision-Language Models
  4. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  5. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Models
  6. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning
  7. 2025Online In-Context Distillation for Low-Resource Vision Language Models
  8. 2025T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs