mlsanityData to 14 Oct 2026

Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering

arXiv 2606.16494 ↗First posted 15 Jun 2026cs.CL cs.AI cs.CVcited by 1

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Knowledge-based visual question answering (KB-VQA) lets vision-language systems answer questions that exceed their parametric knowledge by conditioning a reader on passages retrieved from a Wikipedia-derived knowledge base. In pure-text long-context LLMs, retrieved-context use follows the U-shaped lost-in-the-middle effect of Liu et al. (2024): information at the start and end of context is used, the middle is lost. Whether this transfers to deployed multimodal KB-VQA is open. To close this gap, we design the first controlled probe of reader-side position dependence in multimodal KB-VQA: a gold-position protocol in which only the gold passage's prompt slot varies within question. We run it on three open-source 7B/8B VLM readers and two KB-VQA benchmarks with up to 20 retrieved passages. The shape flips from U to primacy: gold-at-first beats gold-at-last by 16 to 26 points on all six combinations of reader and benchmark, an effect we call Lost at the End; the gap holds at every scale we test, 3B to 32B, attenuating at 32B. Three targeted ablations narrow the cause. A text-only control that removes the image and changes nothing else shows the primacy is already present in text mode and does not depend on the image. Image-position and distractor-shuffle ablations trace the effect to prompt slot 0 of the instruction-tuned reader, where a second answer-bearing passage placed later is largely wasted. On a frozen reader, three retrieval-side fixes (MMR, oracle reranking, rank-based reordering) all fail to improve on the deployment default. Our findings indicate that recall@k is the wrong metric for deployed KB-VQA and that the remaining headroom sits on the reader side; we release our protocol as a controlled instrument for evaluating reader-side interventions.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 19 of its 32 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

19 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation30 May
  2. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs23 Apr
  3. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories15 Jun

Other references · 16

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  6. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 207
  10. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
  11. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 43
  12. 2024PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrieverscited by 42
4 more
  1. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Modelscited by 20
  2. 2026According to Me: Long-Term Personalized Referential Memory QAcited by 14
  3. 2025Do RAG Systems Really Suffer From Positional Bias?cited by 12
  4. 2025Positional Bias in Multimodal Embedding Models: Do They Favor the Beginning, the Middle, or the End?cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
  2. 2026WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
  3. 2026Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
  4. 2025Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
  5. 2025Enabling Collaborative Parametric Knowledge Calibration for Retrieval-Augmented Vision Question Answering
  6. 2024Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering
  7. 2024Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models
  8. 2024GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering