mlsanityData to 7 Oct 2026

Law of Vision Representation in MLLMs

arXiv 2408.16357 ↗First posted 29 Aug 2024cs.CVcited by 16

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present the "Law of Vision Representation" in multimodal large language models (MLLMs). It reveals a strong correlation between the combination of cross-modal alignment, correspondence in vision representation, and MLLM performance. We quantify the two factors using the cross-modal Alignment and Correspondence score (AC score). Through extensive experiments involving thirteen different vision representation settings and evaluations across eight benchmarks, we find that the AC score is linearly correlated to model performance. By leveraging this relationship, we are able to identify and train the optimal vision representation only, which does not require finetuning the language model every time, resulting in a 99.7% reduction in computational cost.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 72 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

59 of 72 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  2. 2024EVLM: An Efficient Vision-Language Model for Visual Understanding19 Jul
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  5. 2024InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understanding3 Mar
  6. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  8. 2023VILA: On Pre-training for Visual Language Models12 Dec
  9. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  12. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
35 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  10. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  11. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  12. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  13. 2023LightGlue: Local Feature Matching at Light Speedcited by 583
  14. 2023Emergent Correspondence from Image Diffusioncited by 437
  15. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  16. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  17. 2020The Lipschitz Constant of Self-Attentioncited by 171
  18. 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 140
  19. 2019Anomaly Detection in Video Sequence with Appearance-Motion Correspondencecited by 118
  20. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  21. 2023Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencecited by 85
  22. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  23. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  24. 2022Multi-modal Alignment using Representation Codebookcited by 63
  25. 2024Focus Anywhere for Fine-grained Multi-page Document Understandingcited by 62
  26. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 46
  27. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46
  28. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attentioncited by 44
  29. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  30. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perceptioncited by 32
  31. 20233DiffTection: 3D Object Detection with Geometry-Aware Diffusion Featurescited by 26
  32. 2024ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Modelscited by 26
  33. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Modelcited by 21
  34. 2024RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulationcited by 20
  35. 2024Unified Lexical Representation for Interpretable Visual-Language Alignmentcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 9; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 34
  3. 2025HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Modelcited by 29
  4. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  5. 2025LanP: Rethinking the Impact of Language Priors in Large Vision-Language Modelscited by 14
  6. 2025MIEB: Massive Image Embedding Benchmarkcited by 11
  7. 2024Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testingcited by 10
  8. 2025LangBridge: Interpreting Image as a Combination of Language Embeddingscited by 9
  9. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Modelscited by 8
  10. 2024Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusioncited by 4
  11. 2025The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insightscited by 3
  12. 2026A Comprehensive Information-Decomposition Analysis of Large Vision-Language Modelscited by 3
4 more
  1. 2024GePBench: Evaluating Fundamental Geometric Perception for Multimodal Large Language Modelscited by 1
  2. 2025Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Modelscited by 1
  3. 2025Zero-Shot Scene Understanding with Multimodal Large Language Models for Automated Vehiclescited by 0
  4. 2026Model Specific Task Similarity for Vision Language Model Selection via Layer Conductancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  2. 2025Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
  3. 2026A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models
  4. 2025Visual Representation Alignment for Multimodal Large Language Models
  5. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  6. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  7. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate
  8. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs