mlsanityData to 14 Oct 2026

Visual Representation Alignment for Multimodal Large Language Models

arXiv 2509.07979 ↗First posted 9 Sep 2025cs.CVcited by 41

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) trained with visual instruction tuning have achieved strong performance across diverse tasks, yet they remain limited in vision-centric tasks such as object counting or spatial reasoning. We attribute this gap to the prevailing text-only supervision paradigm, which provides only indirect guidance for the visual pathway and often leads MLLMs to discard fine-grained visual details during training. In this paper, we present VIsual Representation ALignment (VIRAL), a simple yet effective regularization strategy that aligns the internal visual representations of MLLMs with those of pre-trained vision foundation models (VFMs). By explicitly enforcing this alignment, VIRAL enables the model not only to retain critical visual details from the input vision encoder but also to complement additional visual knowledge from VFMs, thereby enhancing its ability to reason over complex visual inputs. Our experiments demonstrate consistent improvements across all tasks on widely adopted multimodal benchmarks. Furthermore, we conduct comprehensive ablation studies to validate the key design choices underlying our framework. We believe this simple finding opens up an important direction for the effective integration of visual information in training MLLMs.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 64 of its 73 references that are in the map. First posted on arXiv 9 Sep 2025.

Builds on

64 of 73 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMs13 Jun
  2. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thought13 Jan
  3. 2025Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs13 Jan
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec

Other references · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Segment Anythingcited by 8,234
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
38 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2024Depth Anything V2cited by 1,422
  5. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  8. 2023A Survey on Multimodal Large Language Modelscited by 909
  9. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  10. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  12. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  13. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  14. 2024The Platonic Representation Hypothesiscited by 382
  15. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  16. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  17. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 301
  18. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  19. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  20. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  21. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  22. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 155
  23. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  24. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  25. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
  26. 2022Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentationcited by 114
  27. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 98
  28. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 95
  29. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 89
  30. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  31. 2021CATs: Cost Aggregation Transformers for Visual Correspondencecited by 77
  32. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 77
  33. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 69
  34. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 68
  35. 2024Cross-modal Information Flow in Multimodal Large Language Modelscited by 67
  36. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 63
  37. 2024PF3plat: Pose-Free Feed-Forward 3D Gaussian Splattingcited by 62
  38. 2024Reconstructive Visual Instruction Tuningcited by 55

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 32.

41 papers.

Sort:

By area:

  1. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformerscited by 24
  2. 2025Emergent Outlier View Rejection in Visual Geometry Grounded Transformerscited by 22
  3. 2026Vision-aligned Latent Reasoning for Multi-modal Large Language Modelcited by 18
  4. 2025CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMscited by 14
  5. 2026ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervisioncited by 6
  6. 2025VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Modelscited by 5
  7. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
  8. 2025LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendationcited by 5
  9. 2026Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decompositioncited by 5
  10. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 4
  11. 2026TORA: Topological Representation Alignment for 3D Shape Assemblycited by 4
  12. 2026Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Modelscited by 3
29 more
  1. 2026Transferability Between Understanding and Generation in Unified Multimodal Modelscited by 3
  2. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  3. 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Modelscited by 2
  4. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understandingcited by 2
  5. 2026No Pose, No Problem in 4D: Feed-Forward Dynamic Gaussians from Unposed Multi-View Videoscited by 2
  6. 2026GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representationscited by 2
  7. 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoningcited by 2
  8. 2025More Than the Final Answer: Improving Visual Extraction and Logical Consistency in Vision-Language Modelscited by 1
  9. 2026Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindnesscited by 1
  10. 2026FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistantscited by 1
  11. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidancecited by 1
  12. 2026UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigmcited by 1
  13. 2026VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervisioncited by 1
  14. 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answeringcited by 1
  15. 2025Align Your Query: Representation Alignment for Multimodality Medical Object Detectioncited by 0
  16. 2026Logit Lens Supervision for Patch-Level Explanations in Vision-Language Modelscited by 0
  17. 2026DenseMLLM: Standard Multimodal LLMs for Dense Predictioncited by 0
  18. 2026Why MLLMs Struggle to Determine Object Orientationscited by 0
  19. 2026Divide and Conquer: Decoupled Representation Alignment for Multimodal World Modelscited by 0
  20. 2026PGT: Procedurally Generated Tasks for improving visual grounding in MLLMscited by 0
  21. 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learningcited by 0
  22. 2026S2COPE: Self-Supervised Concept Discovery via Preference Learningcited by 0
  23. 2026PROSE: Training-Free Egocentric Scene Registration with Vision-Language Modelscited by 0
  24. 2026Mind the Heads: Topological Representation Alignment for Multimodal LLMscited by 0
  25. 2026Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Modelscited by 0
  26. 2026Mitigating Visual Degradation in MLLMs via Spatial-Spectral Visual Anchor Learningcited by 0
  27. 2026From Sharp Eyes to Expert Mind: Internalizing Expert Knowledge in MLLMs for Tampered Text Detectioncited by 0
  28. 2026Where-OPD: Spatially Guided On-Policy Self-Distillation of MLLMs with Synthetic Scenescited by 0
  29. 2026Can Learning to Answer Spatial Questions Improve Novel-View Generation?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
  2. 2024SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs
  3. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  4. 2026Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models
  5. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  6. 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
  7. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  8. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding