mlsanityData to 7 Oct 2026

On the Out-Of-Distribution Generalization of Multimodal Large Language Models

arXiv 2402.06599 ↗First posted 9 Feb 2024cs.CV cs.AIcited by 30

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We investigate the generalization boundaries of current Multimodal Large Language Models (MLLMs) via comprehensive evaluation under out-of-distribution scenarios and domain-specific tasks. We evaluate their zero-shot generalization across synthetic images, real-world distributional shifts, and specialized datasets like medical and molecular imagery. Empirical results indicate that MLLMs struggle with generalization beyond common training domains, limiting their direct application without adaptation. To understand the cause of unreliable performance, we analyze three hypotheses: semantic misinterpretation, visual feature extraction insufficiency, and mapping deficiency. Results identify mapping deficiency as the primary hurdle. To address this problem, we show that in-context learning (ICL) can significantly enhance MLLMs' generalization, opening new avenues for overcoming generalization barriers. We further explore the robustness of ICL under distribution shifts and show its vulnerability to domain shifts, label shifts, and spurious correlation shifts between in-context examples and test data.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 99 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

62 of 99 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Generative Multimodal Models are In-Context Learners20 Dec
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  3. 2023TouchStone: Evaluating Vision-Language Models by Language Models31 Aug
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2019Invariant Risk Minimizationcited by 1,832
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2020In Search of Lost Domain Generalizationcited by 1,042
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  12. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  13. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  14. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  15. 2023Instruction Tuning with GPT-4cited by 676
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  17. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  18. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  19. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  20. 2019Domain Generalization via Model-Agnostic Learning of Semantic Featurescited by 405
  21. 2023Larger language models do in-context learning differentlycited by 386
  22. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  23. 2020Learning to Generate Novel Domains for Domain Generalizationcited by 296
  24. 2022Visual Prompting via Image Inpaintingcited by 248
  25. 2020Deep Domain-Adversarial Image Generation for Domain Generalisationcited by 231
  26. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  27. 2023SegGPT: Segmenting Everything In Contextcited by 193
  28. 2019Learning to Optimize Domain Specific Normalization for Domain Generalizationcited by 160
  29. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  30. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  31. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  32. 2020Efficient Domain Generalization via Common-Specific Low-Rank Decompositioncited by 134
  33. 2019Regularized Learning for Domain Adaptation under Label Shiftscited by 118
  34. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  35. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  36. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  37. 2021The iWildCam 2021 Competition Datasetcited by 77
  38. 2022ZIN: When and How to Learn Invariance Without Environment Partition?cited by 58

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 16; 2026: 5.

30 papers.

Sort:

By area:

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
  4. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  5. 2024A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualizationcited by 31
  6. 2024Auto-Encoding Morph-Tokens for Multimodal LLMcited by 30
  7. 2025AStar: Boosting Multimodal Reasoning with Automated Structured Thinkingcited by 25
  8. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  9. 2024Large Action Models: From Inception to Implementationcited by 18
  10. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 18
  11. 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
  12. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 15
18 more
  1. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  2. 2025Visual RAG: Expanding MLLM visual knowledge without fine-tuningcited by 11
  3. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  4. 2025Efficient Multi-modal Long Context Learning for Training-free Adaptationcited by 7
  5. 2025Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Strongercited by 6
  6. 2025Visual Instruction Bottleneck Tuningcited by 5
  7. 2025Test-Time Distillation for Continual Model Adaptationcited by 5
  8. 2025Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learningcited by 4
  9. 2025Mind the (Data) Gap: Evaluating Vision Systems in Small Data Applicationscited by 3
  10. 2026EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessmentcited by 3
  11. 2024In-context Learning in Presence of Spurious Correlationscited by 2
  12. 2026Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Trainingcited by 2
  13. 2025COUNTS: Benchmarking Object Detectors and Multimodal Large Language Models under Distribution Shiftscited by 1
  14. 2025FABRIC: Framework for Agent-Based Realistic Intelligence Creationcited by 1
  15. 2026Large Multimodal Models as General In-Context Classifierscited by 1
  16. 2025MapFormer: Self-Supervised Learning of Cognitive Maps with Input-Dependent Positional Embeddingscited by 0
  17. 2026Why We Need World Models for AGI: Where LLMs Fail and How World Models May Outperformcited by 0
  18. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning
  2. 2024On Domain-Adaptive Post-Training for Multimodal Large Language Models
  3. 2025On Path to Multimodal Generalist: General-Level and General-Bench
  4. 2025Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation
  5. 2023On the Performance of Multimodal Language Models
  6. 2024The Revolution of Multimodal Large Language Models: A Survey
  7. 2025Generalizing Large Language Model Usability Across Resource-Constrained
  8. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey