# More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG

arXiv: https://arxiv.org/abs/2503.04388 · HTML: https://mlsanity.com/paper/2503.04388/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 12 of 12)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)
- [Steering Large Language Models between Code Execution and Textual Reasoning](https://mlsanity.com/paper/2410.03524/) (2024)
- [SEAM: A Stochastic Benchmark for Multi-Document Tasks](https://mlsanity.com/paper/2406.16086/) (2024)
- [Revisiting Sentence Union Generation as a Testbed for Text Consolidation](https://mlsanity.com/paper/2305.15605/) (2023)
- [TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network](https://mlsanity.com/paper/2411.02617/) (2024)

## Built on by (most cited first, 16 of 16)

- [Hierarchical Document Refinement for Long-context Retrieval-augmented Generation](https://mlsanity.com/paper/2505.10413/) (2025)
- [Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find](https://mlsanity.com/paper/2505.18148/) (2025)
- [Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents](https://mlsanity.com/paper/2608.11888/) (2026)
- [TASR: Training-Free Adaptive Stopping for Iterative Retrieval](https://mlsanity.com/paper/2606.13814/) (2026)
- [Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds](https://mlsanity.com/paper/2506.03100/) (2025)
- [RECON: Reasoning with Condensation for Efficient Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.10448/) (2025)
- [Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneck](https://mlsanity.com/paper/2601.12499/) (2026)
- [RAGExplorer: A Visual Analytics System for the Comparative Diagnosis of RAG Systems](https://mlsanity.com/paper/2601.12991/) (2026)
- [Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length](https://mlsanity.com/paper/2603.22608/) (2026)
- [Are Finer Citations Always Better? Rethinking Granularity for Attributed Generation](https://mlsanity.com/paper/2604.01432/) (2026)
- [Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA](https://mlsanity.com/paper/2604.22239/) (2026)
- [MeVe: A Modular System for Memory Verification and Effective Context Control in Language Models](https://mlsanity.com/paper/2509.01514/) (2025)
- [IterCOMP: Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answering](https://mlsanity.com/paper/2608.13588/) (2026)
- [Is Implicit Knowledge Enough for LLMs? A RAG Approach for Tree-based Structures](https://mlsanity.com/paper/2510.10806/) (2025)
- [The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context](https://mlsanity.com/paper/2608.14838/) (2026)
- [MegaMem: A Retrieval Solution for Ultra-Large Context Windows](https://mlsanity.com/paper/2608.22137/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) enhances the accuracy of Large Language Model (LLM) responses by leveraging relevant external documents during generation. Although previous studies noted that retrieving many documents can degrade performance, they did not isolate how the quantity of documents affects performance while controlling for context length. We evaluate various language models on custom datasets derived from a multi-hop QA task. We keep the context length and position of relevant information constant while varying the number of documents, and find that increasing the document count in RAG settings poses significant challenges for most LLMs, reducing performance by up to 20%. However, Qwen2.5 maintained consistent results across increasing document counts, indicating better multi-document handling capability. Finally, our results indicate that processing multiple documents is a separate challenge from handling long contexts. We also make the datasets and code available: https://github.com/shaharl6000/MoreDocsSameLen .