# Visual Representation Alignment for Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2509.07979 · HTML: https://mlsanity.com/paper/2509.07979/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 64 of its 73 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 64)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [VisionZip: Longer is Better but Not Necessary in Vision Language Models](https://mlsanity.com/paper/2412.04467/) (2024)
- [A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence](https://mlsanity.com/paper/2305.15347/) (2023)

## Built on by (most cited first, 30 of 41)

- [Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers](https://mlsanity.com/paper/2509.18096/) (2025)
- [Emergent Outlier View Rejection in Visual Geometry Grounded Transformers](https://mlsanity.com/paper/2512.04012/) (2025)
- [Vision-aligned Latent Reasoning for Multi-modal Large Language Model](https://mlsanity.com/paper/2602.04476/) (2026)
- [CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2510.12184/) (2025)
- [ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision](https://mlsanity.com/paper/2602.14276/) (2026)
- [VideoPerceiver: Enhancing Fine-Grained Temporal Perception in Video Multimodal Large Language Models](https://mlsanity.com/paper/2511.18823/) (2025)
- [Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models](https://mlsanity.com/paper/2512.15885/) (2025)
- [LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation](https://mlsanity.com/paper/2512.16891/) (2025)
- [Towards Understanding Modality Interaction in Multimodal Language Models via Partial Information Decomposition](https://mlsanity.com/paper/2606.00959/) (2026)
- [Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models](https://mlsanity.com/paper/2603.00431/) (2026)
- [TORA: Topological Representation Alignment for 3D Shape Assembly](https://mlsanity.com/paper/2604.04050/) (2026)
- [Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models](https://mlsanity.com/paper/2604.08456/) (2026)
- [Transferability Between Understanding and Generation in Unified Multimodal Models](https://mlsanity.com/paper/2607.04423/) (2026)
- [Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models](https://mlsanity.com/paper/2512.06281/) (2025)
- [PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models](https://mlsanity.com/paper/2603.00412/) (2026)
- [See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding](https://mlsanity.com/paper/2605.18018/) (2026)
- [No Pose, No Problem in 4D: Feed-Forward Dynamic Gaussians from Unposed Multi-View Videos](https://mlsanity.com/paper/2605.22190/) (2026)
- [GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations](https://mlsanity.com/paper/2606.03180/) (2026)
- [World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning](https://mlsanity.com/paper/2606.03603/) (2026)
- [More Than the Final Answer: Improving Visual Extraction and Logical Consistency in Vision-Language Models](https://mlsanity.com/paper/2512.12487/) (2025)
- [Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness](https://mlsanity.com/paper/2602.19615/) (2026)
- [FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants](https://mlsanity.com/paper/2603.26008/) (2026)
- [Boosting Visual Instruction Tuning with Self-Supervised Guidance](https://mlsanity.com/paper/2604.12966/) (2026)
- [UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm](https://mlsanity.com/paper/2609.27915/) (2026)
- [VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision](https://mlsanity.com/paper/2609.27948/) (2026)
- [Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering](https://mlsanity.com/paper/2609.38177/) (2026)
- [Align Your Query: Representation Alignment for Multimodality Medical Object Detection](https://mlsanity.com/paper/2510.02789/) (2025)
- [Logit Lens Supervision for Patch-Level Explanations in Vision-Language Models](https://mlsanity.com/paper/2602.01530/) (2026)
- [DenseMLLM: Standard Multimodal LLMs for Dense Prediction](https://mlsanity.com/paper/2602.14134/) (2026)
- [Why MLLMs Struggle to Determine Object Orientations](https://mlsanity.com/paper/2604.13321/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) trained with visual instruction tuning have achieved strong performance across diverse tasks, yet they remain limited in vision-centric tasks such as object counting or spatial reasoning. We attribute this gap to the prevailing text-only supervision paradigm, which provides only indirect guidance for the visual pathway and often leads MLLMs to discard fine-grained visual details during training. In this paper, we present VIsual Representation ALignment (VIRAL), a simple yet effective regularization strategy that aligns the internal visual representations of MLLMs with those of pre-trained vision foundation models (VFMs). By explicitly enforcing this alignment, VIRAL enables the model not only to retain critical visual details from the input vision encoder but also to complement additional visual knowledge from VFMs, thereby enhancing its ability to reason over complex visual inputs. Our experiments demonstrate consistent improvements across all tasks on widely adopted multimodal benchmarks. Furthermore, we conduct comprehensive ablation studies to validate the key design choices underlying our framework. We believe this simple finding opens up an important direction for the effective integration of visual information in training MLLMs.