mlsanityData to 14 Oct 2026

Link-Context Learning for Multimodal LLMs

arXiv 2308.07891 ↗First posted 15 Aug 2023cs.CV cs.CLcited by 24

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The ability to learn from context with novel concepts, and deliver appropriate responses are essential in human conversations. Despite current Multimodal Large Language Models (MLLMs) and Large Language Models (LLMs) being trained on mega-scale datasets, recognizing unseen images or understanding novel concepts in a training-free manner remains a challenge. In-Context Learning (ICL) explores training-free few-shot learning, where models are encouraged to ``learn to learn" from limited tasks and generalize to unseen tasks. In this work, we propose link-context learning (LCL), which emphasizes "reasoning from cause and effect" to augment the learning capabilities of MLLMs. LCL goes beyond traditional ICL by explicitly strengthening the causal relationship between the support set and the query set. By providing demonstrations with causal links, LCL guides the model to discern not only the analogy but also the underlying causal associations between data points, which empowers MLLMs to recognize unseen images and understand novel concepts more effectively. To facilitate the evaluation of this novel approach, we introduce the ISEKAI dataset, comprising exclusively of unseen generated image-label pairs designed for link-context learning. Extensive experiments show that our LCL-MLLM exhibits strong link-context learning capabilities to novel concepts over vanilla MLLMs. Code and data will be released at https://github.com/isekai-portal/Link-Context-Learning.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 37 references that are in the map. First posted on arXiv 15 Aug 2023.

Builds on

30 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  3. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
15 more
  1. 2022Visual Prompt Tuningcited by 1,681
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  4. 2021Grounded Language-Image Pre-trainingcited by 1,156
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2023Instruction Tuning with GPT-4cited by 686
  8. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  9. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  10. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  11. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  12. 2022Unified Vision and Language Prompt Learningcited by 148
  13. 2023Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognitioncited by 35
  14. 2022Prompt Tuning for Generative Multimodal Pretrained Modelscited by 25
  15. 2023Deeply Coupled Cross-Modal Prompt Learningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 9; 2026: 5.

24 papers.

Sort:

By area:

  1. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 154
  2. 2024What Makes Multimodal In-Context Learning Work?cited by 56
  3. 2024Graphic Design with Large Multimodal Modelcited by 40
  4. 2025MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Modelscited by 29
  5. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  6. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  7. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  8. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 10
  9. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  10. 2024Making Large Vision Language Models to be Good Few-shot Learnerscited by 7
  11. 2025Retrieval-augmented in-context learning for multimodal large language models in disease classificationcited by 5
  12. 2026Fine-grained Image Aesthetic Assessment: Learning Discriminative Scores from Relative Rankscited by 5
12 more
  1. 2025MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretrainingcited by 3
  2. 2025CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environmentscited by 3
  3. 2024NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects?cited by 2
  4. 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
  5. 2026Beyond Perceptual Shortcuts: Causal-Inspired Debiasing Optimization for Generalizable Video Reasoning in Lightweight MLLMscited by 2
  6. 2025REF-VLM: Triplet-Based Referring Paradigm for Unified Visual Decodingcited by 1
  7. 2025Classifying the Unknown: In-Context Learning for Open-Vocabulary Text and Symbol Recognitioncited by 1
  8. 2023A Graph-to-Text Approach to Knowledge-Grounded Response Generation in Human-Robot Interactioncited by 0
  9. 2025Vision-Language In-Context Learning Driven Few-Shot Visual Inspection Modelcited by 0
  10. 2025Causality-Driven Infrared and Visible Image Fusioncited by 0
  11. 2026In-Context Collapse in Vision-Language Models and How to Mitigate it?cited by 0
  12. 2026When Is a Task Vector Enough? An Empirical Theory of Implicit Multimodal ICLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FewMMBench: A Benchmark for Multimodal Few-Shot Learning
  2. 2024Towards Multimodal In-Context Learning for Vision & Language Models
  3. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
  4. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  5. 2024Can MLLMs Perform Text-to-Image In-Context Learning?
  6. 2024Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
  7. 2025What do vision-language models see in the context? Investigating multimodal in-context learning
  8. 2026GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models