mlsanityData to 7 Oct 2026

GRAMMAR: Grounded and Modular Methodology for Assessment of Closed-Domain Retrieval-Augmented Language Model

arXiv 2404.19232 ↗First posted 30 Apr 2024cs.CL cs.AIcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems are widely used across various industries for querying closed-domain and in-house knowledge bases. However, evaluating these systems presents significant challenges due to the private nature of closed-domain data and a scarcity of queries with verifiable ground truths. Moreover, there is a lack of analytical methods to diagnose problematic modules and identify types of failure, such as those caused by knowledge deficits or issues with robustness. To address these challenges, we introduce GRAMMAR (GRounded And Modular Methodology for Assessment of RAG), an evaluation framework comprising a grounded data generation process and an evaluation protocol that effectively pinpoints defective modules. Our validation experiments reveal that GRAMMAR provides a reliable approach for identifying vulnerable modules and supports hypothesis testing for textual form vulnerabilities. An open-source tool accompanying this framework is available in our GitHub repository (see https://github.com/xinzhel/grammar), allowing for easy reproduction of our results and enabling reliable and modular evaluation in closed-domain settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 14 of its 29 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

14 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 13

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  4. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  5. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  6. 2023Whose Opinions Do Language Models Reflect?cited by 759
  7. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  8. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  9. 2018Generating Natural Language Adversarial Examplescited by 422
  10. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  11. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  12. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 118
1 more
  1. 2021From LSAT: The Progress and Challenges of Complex Reasoningcited by 43

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  2. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
  3. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  5. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  6. 2025Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation
  7. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation
  8. 2025OmniBench-RAG: A Multi-Domain Evaluation Platform for Retrieval-Augmented Generation Tools