The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG
also in 1 more
Abstract (arXiv)
While Multimodal Large Language Models (MLLMs) are increasingly integrated with Retrieval-Augmented Generation (RAG) to mitigate hallucinations, the introduction of external documents can conceal severe failure modes at the instance level. We identify and formalize the phenomenon of recorruption, where the introduction of even perfectly accurate "oracle" context causes a capable model to abandon an initially correct prediction. Through a mechanistic diagnosis of internal attention matrices, we show that recorruption is driven by a two-fold attentional collapse: (1) visual blindness, characterized by the systemic suppression of visual attention mass (Mvis) and sharpness (Svis), and (2) a structural positional bias that forces the model to prioritize boundary tokens over semantic relevance. Our analysis reveals an Illusion of Success, demonstrating that many seemingly correct RAG outcomes are merely positional coincidences where the model's textual copying bias happens to align with the ground-truth location. To address these vulnerabilities, we propose Bottleneck Attention Intervention for Recovery (BAIR), a parameter-free, inference-time framework that restores visual saliency and applies position-aware penalties to textual distractors. Across medical factuality, social fairness, and geospatial benchmarks, BAIR successfully restores multimodal grounding and improves diagnostic reliability without requiring model retraining or fine-tuning.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 29 references that are in the map. First posted on arXiv 7 May 2026.
Builds on
24 of 29 references.
Closest ancestors · 1
Same direction, newest first
- 2026When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs30 Jan
Other references · 23
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
- 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
- 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
- 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
- 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
- 2025MedGemma Technical Reportcited by 436
- 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
- 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 104
- 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
11 more
- 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 85
- 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
- 2024CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answeringcited by 72
- 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
- 2025On the Emergence of Position Bias in Transformerscited by 59
- 2023FACET: Fairness in Computer Vision Evaluation Benchmarkcited by 58
- 2024Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documentscited by 27
- 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 24
- 2024From RAGs to rich parameters: Probing how language models utilize external knowledge over parametric information for factual queriescited by 16
- 2025Automated Structured Radiology Report Generationcited by 15
- 2025Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generationcited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content
- 2026When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models
- 2023Attention Sorting Combats Recency Bias In Long Context Language Models
- 2026Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models
- 2025Re-ranking the Context for Multimodal Retrieval Augmented Generation
- 2026Disentangling Semantic Attention from Structural Bias in the Attention Manifold
- 2025FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
- 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performance