mlsanityData to 14 Oct 2026

Visual Instruction Bottleneck Tuning

arXiv 2505.13946 ↗First posted 20 May 2025cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Despite widespread adoption, multimodal large language models (MLLMs) suffer performance degradation when encountering unfamiliar queries under distribution shifts. Existing methods to improve MLLM generalization typically require either more instruction data or larger advanced model architectures, both of which incur non-trivial human labor or computational costs. In this work, we take an alternative approach to enhance the generalization and robustness of MLLMs under distribution shifts, from a representation learning perspective. Inspired by information bottleneck (IB) principle, we derive a variational lower bound of the IB for MLLMs and devise a practical implementation, Visual Instruction Bottleneck Tuning (Vittle). We then provide a theoretical justification of Vittle by revealing its connection to an information-theoretic robustness metric of MLLM. Empirical validation of multiple MLLMs on open-ended and closed-form question answering and object hallucination detection tasks over 45 datasets, including 30 shift scenarios, demonstrates that Vittle consistently improves the MLLM's robustness under shifts by pursuing the learning of a minimal sufficient representation.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 95 of its 134 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

95 of 134 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach1 Feb
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data24 Oct
  4. 2024Reconstructive Visual Instruction Tuning12 Oct
  5. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  6. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  7. 2024MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs24 Jun
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  9. 2024Towards Robust Instruction Tuning on Multimodal Large Language Models22 Feb
  10. 2024The Revolution of Multimodal Large Language Models: A Survey19 Feb
  11. 2024On the Out-Of-Distribution Generalization of Multimodal Large Language Models9 Feb
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
3 more
  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 80

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
38 more
  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  5. 2016Variational Inference: A Review for Statisticianscited by 1,472
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  9. 2016Deep Variational Information Bottleneckcited by 930
  10. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  11. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  12. 2021Robust fine-tuning of zero-shot modelscited by 797
  13. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  14. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  15. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  16. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  17. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  18. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  19. 2020CLUB: A Contrastive Log-ratio Upper Bound of Mutual Informationcited by 235
  20. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  21. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
  22. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 191
  23. 2022Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanismcited by 171
  24. 2020Graph Information Bottleneckcited by 168
  25. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 166
  26. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  27. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 151
  28. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
  29. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  30. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  31. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 132
  32. 2020Learning to Learn with Variational Information Bottleneck for Domain Generalizationcited by 102
  33. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  34. 2023How Does Information Bottleneck Help Deep Learning?cited by 93
  35. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attackscited by 92
  36. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  37. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  38. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 75

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  2. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 14
  3. 2025UniGame: Turning a Unified Multimodal Model Into Its Own Adversarycited by 9
  4. 2026VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognitioncited by 0
  5. 2026StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering
  2. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
  3. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  4. 2025Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation
  5. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning
  6. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly
  7. 2023On the Performance of Multimodal Language Models
  8. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning