mlsanityData to 14 Oct 2026

PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation

arXiv 2507.22927 ↗First posted 23 Jul 2025cs.CLcited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge, where the LLM's ability to generate responses based on the combination of a given query and retrieved documents is crucial. However, most benchmarks focus on overall RAG system performance, rarely assessing LLM-specific capabilities. Current benchmarks emphasize broad aspects such as noise robustness, but lack a systematic and granular evaluation framework on document utilization. To this end, we introduce textitPlaceholder-RAG-Benchmark, a multi-level fine-grained benchmark, emphasizing the following progressive dimensions: (1) multi-level filtering abilities, (2) combination abilities, and (3) reference reasoning. To provide a more nuanced understanding of LLMs' roles in RAG systems, we formulate an innovative placeholder-based approach to decouple the contributions of the LLM's parametric knowledge and the external knowledge. Experiments demonstrate the limitations of representative LLMs in the RAG system's generation capabilities, particularly in error resilience and context faithfulness. Our benchmark provides a reproducible framework for developing more reliable and efficient RAG systems. Our code is available in https://github.com/Alipay-Med/PRGB.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 12 of its 13 references that are in the map. First posted on arXiv 23 Jul 2025.

Builds on

12 of 13 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation8 Jul
  2. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems25 Jun
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 9

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  6. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  7. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  8. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  9. 2024Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answeringcited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025DIVER: A Multi-Stage Approach for Reasoning-intensive Information Retrievalcited by 37
  2. 2026CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
  2. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Level
  3. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  4. 2025Evaluating the Retrieval Robustness of Large Language Models
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  6. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning
  7. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generation
  8. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs