mlsanityData to 14 Oct 2026

Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach

arXiv 2502.00577 ↗First posted 1 Feb 2025cs.AI cs.CL cs.LGcited by 16

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) have shown promising capabilities but struggle under distribution shifts, where evaluation data differ from instruction tuning distributions. Although previous works have provided empirical evaluations, we argue that establishing a formal framework that can characterize and quantify the risk of MLLMs is necessary to ensure the safe and reliable application of MLLMs in the real world. By taking an information-theoretic perspective, we propose the first theoretical framework that enables the quantification of the maximum risk of MLLMs under distribution shifts. Central to our framework is the introduction of Effective Mutual Information (EMI), a principled metric that quantifies the relevance between input queries and model responses. We derive an upper bound for the EMI difference between in-distribution (ID) and out-of-distribution (OOD) data, connecting it to visual and textual distributional discrepancies. Extensive experiments on real benchmark datasets, spanning 61 shift scenarios, empirically validate our theoretical insights.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 80 references that are in the map. First posted on arXiv 1 Feb 2025.

Builds on

56 of 80 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024Adapting Large Multimodal Models to Distribution Shifts: The Role of In-Context Learning20 May
  3. 2024On the Out-Of-Distribution Generalization of Multimodal Large Language Models9 Feb
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
38 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  3. 2016Deep Variational Information Bottleneckcited by 930
  4. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  5. 2021Robust fine-tuning of zero-shot modelscited by 797
  6. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  8. 2018MINE: Mutual Information Neural Estimationcited by 683
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  10. 2019On Variational Bounds of Mutual Informationcited by 559
  11. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  12. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  13. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
  14. 2019On Mutual Information Maximization for Representation Learningcited by 316
  15. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 235
  16. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
  17. 2020Learning Deep Kernels for Non-Parametric Two-Sample Testscited by 130
  18. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 126
  19. 2016R'enyi Divergence Variational Inferencecited by 104
  20. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  21. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 89
  22. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  23. 2021FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoderscited by 75
  24. 2021Conditional probing: measuring usable information beyond a baselinecited by 62
  25. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  26. 2024Transformers are Universal In-context Learnerscited by 43
  27. 2023Trainable Projected Gradient Method for Robust Fine-tuningcited by 37
  28. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  29. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  30. 2018Robustness of Conditional GANs to Noisy Labelscited by 27
  31. 2021An Information-theoretic Approach to Distribution Shiftscited by 25
  32. 2021Neural Estimation of Statistical Divergencescited by 25
  33. 2022REV: Information-Theoretic Evaluation of Free-Text Rationalescited by 22
  34. 2023How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigationcited by 22
  35. 2023Robust Prompt Optimization for Large Language Models Against Distribution Shiftscited by 19
  36. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  37. 2023The Representation Jensen-Shannon Divergencecited by 9
  38. 2025Aligning Instruction Tuning with Pre-trainingcited by 7

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 10.

16 papers.

Sort:

By area:

  1. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  2. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 14
  3. 2025SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularizationcited by 12
  4. 2025Failure Modes in LLM Systems: A System-Level Taxonomy for Reliable AI Applicationscited by 10
  5. 2025UniGame: Turning a Unified Multimodal Model Into Its Own Adversarycited by 9
  6. 2025Visual Instruction Bottleneck Tuningcited by 5
  7. 2026A Comprehensive Information-Decomposition Analysis of Large Vision-Language Modelscited by 4
  8. 2026Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Schedulingcited by 2
  9. 2026Are Video Reasoning Models Ready to Go Outside?cited by 2
  10. 2025ICYM2I: The illusion of multimodal informativeness under missingnesscited by 1
  11. 2026Beyond In-Domain Detection: SpikeScore for Cross-Domain Hallucination Detectioncited by 1
  12. 2026NeuroVLM-Bench: Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorderscited by 1
4 more
  1. 2026Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathologycited by 1
  2. 2026CoDA: Towards Effective Cross-domain Knowledge Transfer via CoT-guided Domain Adaptationcited by 0
  3. 2026COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Samplingcited by 0
  4. 2026Local Sparsity Enables Unsupervised LLM Safety Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating Stochastic Collapse and Implicit Bias in Multimodal Large Language Models
  2. 2025Do Large Language Models (Really) Need Statistical Foundations?
  3. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models
  5. 2026Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs
  6. 2025Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression
  7. 2025Self-Improvement in Multimodal Large Language Models: A Survey
  8. 2025LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models