mlsanityData to 7 Oct 2026

RAVE: Re-Allocating Visual Attention in Large Multimodal Models

arXiv 2605.18359 ↗First posted 18 May 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models (LMMs) inherit the self-attention mechanism of pretrained language backbones, yet standard attention can exhibit suboptimal allocation, including cross-modal misallocation between textual and visual evidence and intra-visual imbalance among visual tokens. We propose RAVE (Re-Allocating Visual Attention), a lightweight pair-gating mechanism that adds a learned query-key bias to pre-softmax attention scores over visual keys, derived from pre-RoPE query and key features. RAVE requires no architectural modification to the backbone and can be trained end-to-end with the rest of the model. Across a suite of multimodal benchmarks, RAVE improves over standard attention by an average of 3 points, with the largest gains on perception-intensive tasks -- including multilingual OCR, chart understanding, document VQA, and scene text VQA -- where accurate visual grounding is critical.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 40 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

33 of 40 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Qwen3-VL Technical Report26 Nov
  2. 2025Rethinking Visual Information Processing in Multimodal LLMs13 Nov
  3. 2025MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding7 Jul
  4. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs4 Mar
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  6. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 26

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
14 more
  1. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  2. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  3. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  4. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  5. 2021InfographicVQAcited by 430
  6. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  7. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  8. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  9. 2025Look-Back: Implicit Visual Re-focusing in MLLM Reasoningcited by 54
  10. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  11. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  12. 2025Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoningcited by 28
  13. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  14. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ChartDensity-Bench: Benchmarking MLLMs for Numerical Data Reconstruction under Visual Densitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance
  2. 2026Disentangling Semantic Attention from Structural Bias in the Attention Manifold
  3. 2026Learning to See What You Need: Gaze Attention for Multimodal Large Language Models
  4. 2026VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes
  5. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
  6. 2025Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis
  7. 2026LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models
  8. 2025SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs