mlsanityData to 7 Oct 2026

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

arXiv 2603.06198 ↗First posted 6 Mar 2026cs.CLcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) is a framework in which a Generator, such as a Large Language Model (LLM), produces answers by retrieving documents from an external collection using a Retriever. In practice, Generators must integrate evidence from long contexts, perform multi-step reasoning, interpret tables, and abstain when evidence is missing. However, existing benchmarks for Generators provide limited coverage, with none enabling simultaneous evaluation of multiple capabilities under unified conditions. To bridge the gap between existing evaluations and practical use, we introduce LIT-RAGBench (the Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark), which defines five categories: Integration, Reasoning, Logic, Table, and Abstention, each further divided into practical evaluation aspects. LIT-RAGBench systematically covers patterns combining multiple aspects across categories. By using fictional entities and scenarios, LIT-RAGBench evaluates answers grounded in the provided external documents. The dataset consists of 114 human-constructed Japanese questions and an English version generated by machine translation with human curation. We use LLM-as-a-Judge for scoring and report category-wise and overall accuracy. Across API-based and open-weight models, no model exceeds 90% overall accuracy. By making strengths and weaknesses measurable within each category, LIT-RAGBench serves as a valuable metric for model selection in practical RAG deployments and for building RAG-specialized models. We release LIT-RAGBench, including the dataset and evaluation code, at https://github.com/Koki-Itai/LIT-RAGBench.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 19 of its 29 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

19 of 29 references.

References · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  8. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  9. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 242
  10. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  11. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  12. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
7 more
  1. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  2. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  3. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  4. 2024Efficacy of Synthetic Data as a Benchmarkcited by 29
  5. 2023Large Language Models are Complex Table Parserscited by 28
  6. 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 25
  7. 2025What Has Been Lost with Synthetic Evaluation?cited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  4. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  5. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation
  6. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
  7. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models
  8. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions