mlsanityData to 14 Oct 2026

Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models

arXiv 2512.06281 ↗First posted 6 Dec 2025cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in multimodal tasks. Despite their impressive performance, MLLMs suffer from the modality imbalance issue, where visual information is often underutilized compared to textual representations in deeper layers, leading to degraded visual performance or hallucinations. This issue stems from the predominant reliance on next-text-token-prediction during training, which fails to provide direct visual supervisory signals, resulting in progressive homogenization of visual representations throughout the layers. To this end, we propose Latent Visual Reconstruction (LaVer), a novel training framework that facilitates MLLMs in learning more discriminative visual representations via masked image modeling in the joint latent semantic space of LLM. Our method offers direct visual activation to MLLMs, which exhibit increased visual attention allocation, indicating enhanced utilization of visual information. Extensive experiments across diverse benchmarks prove the superiority of our approach in various scenarios, especially those requiring dense visual capabilities. Code of LaVer is available at https://github.com/Fir-lat/LaVer.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 118 of its 130 references that are in the map. First posted on arXiv 6 Dec 2025.

Builds on

118 of 130 references.

Closest ancestors · 30

Same direction, newest first

  1. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training28 Sep
  2. 2025Visual Representation Alignment for Multimodal Large Language Models9 Sep
  3. 2025When Language Overrules: Revealing Text Dominance in Multimodal Large Language Models14 Aug
  4. 2025How Visual Representations Map to Language Feature Space in Multimodal LLMs13 Jun
  5. 2025MLLMs are Deeply Affected by Modality Bias24 May
  6. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer14 Apr
  7. 2025Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization23 Mar
  8. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices8 Mar
  9. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs24 Feb
  10. 2025Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion6 Feb
  11. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization17 Nov
  12. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding15 Oct
18 more
  1. 2024Reconstructive Visual Instruction Tuning12 Oct
  2. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs10 Oct
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models22 Aug
  5. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  6. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  8. 2024Dense Connector for MLLMs22 May
  9. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective27 Mar
  10. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  11. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization13 Mar
  12. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priors8 Mar
  13. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  14. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  15. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  16. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  17. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  18. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 88

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2025Qwen3 Technical Reportcited by 8,820
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  13. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  14. 2024Qwen2 Technical Reportcited by 2,120
  15. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  16. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  17. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  18. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  19. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  20. 2019Towards VQA Models That Can Readcited by 1,718
  21. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  22. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  23. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  24. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  25. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  26. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  27. 2016Modeling Context in Referring Expressionscited by 1,063
  28. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  29. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  30. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  31. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  32. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  33. 2016A Diagram Is Worth A Dozen Imagescited by 804
  34. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  35. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  36. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  37. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  38. 2018Object Hallucination in Image Captioningcited by 669

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigmcited by 1
  2. 2026VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervisioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  2. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  3. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
  4. 2025Multimodal Language Models See Better When They Look Shallower
  5. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  6. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See
  7. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
  8. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference