mlsanityData to 14 Oct 2026

Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG

arXiv 2410.05983 ↗First posted 8 Oct 2024cs.CL cs.AI cs.LGcited by 123

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-augmented generation (RAG) empowers large language models (LLMs) to utilize external knowledge sources. The increasing capacity of LLMs to process longer input sequences opens up avenues for providing more retrieved information, to potentially enhance the quality of generated outputs. It is plausible to assume that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance. However, our empirical findings demonstrate that for many long-context LLMs, the quality of generated output initially improves first, but then subsequently declines as the number of retrieved passages increases. This paper investigates this phenomenon, identifying the detrimental impact of retrieved "hard negatives" as a key contributor. To mitigate this and enhance the robustness of long-context LLM-based RAG, we propose both training-free and training-based approaches. We first showcase the effectiveness of retrieval reordering as a simple yet powerful training-free optimization. Furthermore, we explore training-based methods, specifically RAG-specific implicit LLM fine-tuning and RAG-oriented fine-tuning with intermediate reasoning, demonstrating their capacity for substantial performance gains. Finally, we conduct a systematic analysis of design choices for these training-based methods, including data distribution, retriever selection, and training context length.

Key facts. Cited by 123 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 39 of its 52 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

39 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach23 Jul
  2. 2024Evaluation of Retrieval-Augmented Generation: A Survey13 May
  3. 2024Corrective Retrieval Augmented Generation29 Jan
  4. 2024The Power of Noise: Redefining Retrieval for RAG Systems26 Jan
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 34

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2023Mistral 7Bcited by 3,164
  7. 2023A Survey of Large Language Modelscited by 2,576
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
22 more
  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  2. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  3. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  4. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  6. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  7. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  8. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  9. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
  10. 2024Many-Shot In-Context Learningcited by 206
  11. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  12. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  13. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  14. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 184
  15. 2021Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answeringcited by 178
  16. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  17. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  18. 2023Retrieval meets Long Context Large Language Modelscited by 115
  19. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
  20. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  21. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  22. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 56

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 77; 2026: 40.

123 papers.

Sort:

By area:

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  3. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 127
  4. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  5. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 88
  6. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  7. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  8. 2025The Distracting Effect: Understanding Irrelevant Passages in RAGcited by 33
  9. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 31
  10. 2025RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learningcited by 30
  11. 2025SGMem: Sentence Graph Memory for Long-Term Conversational Agentscited by 29
  12. 2025MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learningcited by 27
38 more
  1. 2025Large Language Models Miss the Multi-Agent Markcited by 26
  2. 2025ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learningcited by 26
  3. 2025Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Rewardcited by 25
  4. 2024Long Context RAG Performance of Large Language Modelscited by 23
  5. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 23
  6. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  7. 2025Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-kcited by 21
  8. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 20
  9. 2025PISCO: Pretty Simple Compression for Retrieval-Augmented Generationcited by 20
  10. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  11. 2025Benchmarking Retrieval-Augmented Generation for Chemistrycited by 20
  12. 2025CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Executioncited by 14
  13. 2026Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answeringcited by 14
  14. 2024APEX2: Adaptive and Extreme Summarization for Personalized Knowledge Graphscited by 13
  15. 2025TracLLM: A Generic Framework for Attributing Long Context LLMscited by 13
  16. 2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
  17. 2025Do RAG Systems Really Suffer From Positional Bias?cited by 12
  18. 2025AgriGPT: a Large Language Model Ecosystem for Agriculturecited by 11
  19. 2026OpenDecoder: Open Large Language Model Decoding to Incorporate Document Quality in RAGcited by 11
  20. 2024Multi-Reranker: Maximizing performance of retrieval-augmented generation in the FinanceRAG challengecited by 10
  21. 2025RAG-KG-IL: A Multi-Agent Hybrid Framework for Reducing Hallucinations and Enhancing LLM Reasoning through RAG and Incremental Knowledge Graph Learning Integrationcited by 9
  22. 2025ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learningcited by 9
  23. 2025HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reusecited by 8
  24. 2025Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarkscited by 8
  25. 2025EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notescited by 8
  26. 2025ContextPilot: Fast Long-Context Inference via Context Reusecited by 8
  27. 2025Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAGcited by 7
  28. 2025Influence Guided Context Selection for Effective Retrieval-Augmented Generationcited by 7
  29. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 7
  30. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Modelscited by 7
  31. 2025CDF-RAG: Causal Dynamic Feedback for Adaptive Retrieval-Augmented Generationcited by 6
  32. 2026The Context Gathering Decision Process: A POMDP Framework for Agentic Searchcited by 6
  33. 2025Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMscited by 5
  34. 2025FIRESPARQL: A LLM-based Framework for SPARQL Query Generation over Scholarly Knowledge Graphscited by 5
  35. 2025Beyond the limitation of a single query: Train your LLM for query expansion with Reinforcement Learningcited by 5
  36. 2025Query Decomposition for RAG: Balancing Exploration-Exploitationcited by 5
  37. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMscited by 5
  38. 2025Hypercube-Based Retrieval-Augmented Generation for Scientific Question-Answeringcited by 4

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Tuning LLMs by RAG Principles: Towards LLM-native Memory
  2. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  3. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
  4. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing
  5. 2025Post-training an LLM for RAG? Train on Self-Generated Demonstrations
  6. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference
  7. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely
  8. 2024In Defense of RAG in the Era of Long-Context Language Models