mlsanityData to 14 Oct 2026

Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models

arXiv 2505.05626 ↗First posted 8 May 2025cs.CV cs.AIcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Achieving deep alignment between vision and language remains a central challenge for Multimodal Large Language Models (MLLMs). These models often fail to fully leverage visual input, defaulting to strong language priors. Our approach first provides insights into how MLLMs internally build visual understanding of image regions and then introduces techniques to amplify this capability. Specifically, we explore techniques designed both to deepen the model's understanding of visual content and to ensure that these visual insights actively guide language generation. We demonstrate the superior multimodal understanding of our resultant model through a detailed upstream analysis quantifying its ability to predict visually-dependent tokens as well as 10 pt boost on visually challenging tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 22 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

21 of 22 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models24 Jan
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan

Other references · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  5. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  6. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  7. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  8. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  9. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  10. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  11. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  12. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 113
5 more
  1. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 109
  2. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  3. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Modelscited by 50
  4. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 34
  5. 2022From colouring-in to pointillism: revisiting semantic segmentation supervisioncited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Mapscited by 23
  2. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 12
  3. 2025VideoAnchor: Reinforcing Subspace-Structured Visual Cues for Coherent Visual-Spatial Reasoningcited by 1
  4. 2026Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach
  2. 2025Visual Representation Alignment for Multimodal Large Language Models
  3. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs
  4. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  5. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  6. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  7. 2026Beyond Language Modeling: An Exploration of Multimodal Pretraining
  8. 2026Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention