# Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2506.11063 · HTML: https://mlsanity.com/paper/2506.11063/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 30 May 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 13 of 13)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [DocPrompting: Generating Code by Retrieving the Docs](https://mlsanity.com/paper/2207.05987/) (2022)
- [ListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrieval](https://mlsanity.com/paper/2402.15838/) (2024)
- [Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2410.16983/) (2024)
- [UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation](https://mlsanity.com/paper/2408.11305/) (2024)
- [Can Graph Descriptive Order Affect Solving Graph Problems with LLMs?](https://mlsanity.com/paper/2402.07140/) (2024)
- [Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval](https://mlsanity.com/paper/2404.04163/) (2024)
- [VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models](https://mlsanity.com/paper/2406.10228/) (2024)
- [SoccerRAG: Multimodal Soccer Information Retrieval via Natural Queries](https://mlsanity.com/paper/2406.01273/) (2024)

## Built on by (most cited first, 9 of 9)

- [Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning](https://mlsanity.com/paper/2509.06461/) (2025)
- [Not in Sync: Unveiling Temporal Bias in Audio Chat Models](https://mlsanity.com/paper/2510.12185/) (2025)
- [Revealing the Attention Floating Mechanism in Masked Diffusion Models](https://mlsanity.com/paper/2601.07894/) (2026)
- [MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation](https://mlsanity.com/paper/2509.24253/) (2025)
- [Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization](https://mlsanity.com/paper/2605.11974/) (2026)
- [Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering](https://mlsanity.com/paper/2606.16494/) (2026)
- [A2R2: Advancing Img2LaTeX Conversion via Visual Reasoning with Attention-Guided Refinement](https://mlsanity.com/paper/2507.20890/) (2025)
- [The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.05594/) (2026)
- [Retrieval Sensitivity to Identity Signals in Queries](https://mlsanity.com/paper/2609.36534/) (2026)

## Abstract (arXiv)

Multimodal Retrieval-Augmented Generation (RAG) systems have become essential in knowledge-intensive and open-domain tasks. As retrieval complexity increases, ensuring the robustness of these systems is critical. However, current RAG models are highly sensitive to the order in which evidence is presented, often resulting in unstable performance and biased reasoning, particularly as the number of retrieved items or modality diversity grows. This raises a central question: How does the position of retrieved evidence affect multimodal RAG performance? To answer this, we present the first comprehensive study of position bias in multimodal RAG systems. Through controlled experiments across text-only, image-only, and mixed-modality tasks, we observe a consistent U-shaped accuracy curve with respect to evidence position. To quantify this bias, we introduce the Position Sensitivity Index (PSIp) and develop a visualization framework to trace attention allocation patterns across decoder layers. Our results reveal that multimodal interactions intensify position bias compared to unimodal settings, and that this bias increases logarithmically with retrieval range. These findings offer both theoretical and empirical foundations for position-aware analysis in RAG, highlighting the need for evidence reordering or debiasing strategies to build more reliable and equitable generation systems.