mlsanityData to 7 Oct 2026

Benchmarking Large Language Models in Retrieval-Augmented Generation

arXiv 2309.01431 ↗First posted 4 Sep 2023cs.CLcited by 307

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented generation on different large language models, which make it challenging to identify the potential bottlenecks in the capabilities of RAG for different LLMs. In this paper, we systematically investigate the impact of Retrieval-Augmented Generation on large language models. We analyze the performance of different large language models in 4 fundamental abilities required for RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness. To this end, we establish Retrieval-Augmented Generation Benchmark (RGB), a new corpus for RAG evaluation in both English and Chinese. RGB divides the instances within the benchmark into 4 separate testbeds based on the aforementioned fundamental abilities required to resolve the case. Then we evaluate 6 representative LLMs on RGB to diagnose the challenges of current LLMs when applying RAG. Evaluation reveals that while LLMs exhibit a certain degree of noise robustness, they still struggle significantly in terms of negative rejection, information integration, and dealing with false information. The aforementioned assessment outcomes indicate that there is still a considerable journey ahead to effectively apply RAG to LLMs.

Key facts. Cited by 307 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 36 of its 51 references that are in the map. First posted on arXiv 4 Sep 2023.

Builds on

36 of 51 references.

References · 36

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
24 more
  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  2. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  3. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  5. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  6. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  7. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  8. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  9. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  10. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  11. 2022Large Language Models with Controllable Working Memorycited by 169
  12. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  13. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  14. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  15. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 146
  16. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  17. 2023Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecturecited by 119
  18. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  19. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  20. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  21. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 80
  22. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  23. 2018Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memorycited by 49
  24. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 118; 2025: 106; 2026: 74.

307 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  4. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  5. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  6. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  7. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  8. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  9. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  10. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation Systemcited by 138
  11. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  12. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
38 more
  1. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  2. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  3. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  4. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 109
  5. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  6. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  7. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  8. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  9. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
  10. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 88
  11. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
  12. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  13. 2024StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurizationcited by 70
  14. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  15. 2024KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generationcited by 67
  16. 2024Think-on-Graph 2.0: Deep and Faithful Large Language Model Reasoning with Knowledge-guided Retrieval Augmented Generationcited by 63
  17. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  18. 2025Retrieval-Augmented Generation with Conflicting Evidencecited by 60
  19. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59
  20. 2024Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Modelscited by 57
  21. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 51
  22. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systemscited by 50
  23. 2024How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?cited by 44
  24. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  25. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysiscited by 40
  26. 2025Practical Poisoning Attacks against Retrieval-Augmented Generationcited by 35
  27. 2024SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agentscited by 34
  28. 2024Generative AI Agents with Large Language Model for Satellite Networks via a Mixture of Experts Transmissioncited by 34
  29. 2024RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewardscited by 33
  30. 2024Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuningcited by 31
  31. 2024LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scalecited by 30
  32. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 30
  33. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  34. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 29
  35. 2024PersonaRAG: Enhancing Retrieval-Augmented Generation Systems with User-Centric Agentscited by 28
  36. 2025Traceback of Poisoning Attacks to Retrieval-Augmented Generationcited by 28
  37. 2024Exploring Large Language Models for Feature Selection: A Data-centric Perspectivecited by 27
  38. 2025RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defectscited by 27

50 most cited of 307 All 307 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models
  2. 2025PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation
  3. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  4. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey
  5. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
  6. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  7. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
  8. 2025Less LLM, More Documents: Searching for Improved RAG