mlsanityData to 14 Oct 2026

Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework

arXiv 2411.09607 ↗First posted 14 Nov 2024cs.IR cs.CLcited by 28

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

This report provides an initial look at partial results from the TREC 2024 Retrieval-Augmented Generation (RAG) Track. We have identified RAG evaluation as a barrier to continued progress in information access (and more broadly, natural language processing and artificial intelligence), and it is our hope that we can contribute to tackling the many challenges in this space. The central hypothesis we explore in this work is that the nugget evaluation methodology, originally developed for the TREC Question Answering Track in 2003, provides a solid foundation for evaluating RAG systems. As such, our efforts have focused on "refactoring" this methodology, specifically applying large language models to both automatically create nuggets and to automatically assign nuggets to system answers. We call this the AutoNuggetizer framework. Within the TREC setup, we are able to calibrate our fully automatic process against a manual process whereby nuggets are created by human assessors semi-manually and then assigned manually to system answers. Based on initial results across 21 topics from 45 runs, we observe a strong correlation between scores derived from a fully automatic nugget evaluation and a (mostly) manual nugget evaluation by human assessors. This suggests that our fully automatic evaluation process can be used to guide future iterations of RAG systems.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 9 of its 18 references that are in the map. First posted on arXiv 14 Nov 2024.

Builds on

9 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track24 Jun

Other references · 8

  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  3. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 155
  4. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  5. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 61
  6. 2024UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessorcited by 42
  7. 2024Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agentscited by 23
  8. 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Lookcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 13.

28 papers.

Sort:

By area:

  1. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  2. 2025Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generationcited by 15
  3. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  4. 2025Benchmarking LLM-based Relevance Judgment Methodscited by 14
  5. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judgescited by 14
  6. 2025Auto-ARGUE: LLM-Based Report Generation Evaluationcited by 14
  7. 2025Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggetscited by 8
  8. 2025RAVine: Reality-Aligned Evaluation for Agentic Searchcited by 6
  9. 2025All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generationscited by 6
  10. 2025An Efficient Rubric-based Generative Verifier for Search-Augmented LLMscited by 4
  11. 2025The Viability of Crowdsourcing for RAG Evaluationcited by 2
  12. 2025LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiabilitycited by 2
16 more
  1. 2026Incorporating Q&A Nuggets into Retrieval-Augmented Generationcited by 2
  2. 2026LANCER: LLM Reranking for Nugget Coveragecited by 2
  3. 2026CoverageBench: Evaluating Information Coverage across Tasks and Domainscited by 2
  4. 2025UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responsescited by 1
  5. 2026Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?cited by 1
  6. 2026Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coveragecited by 1
  7. 2026Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelinescited by 1
  8. 2026NuggetIndex: Governed Atomic Retrieval for Maintainable RAGcited by 1
  9. 2026DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluationcited by 1
  10. 2026QO-Bench: Diagnosing Query-Operator-Preserving Retrieval over Typed Event Tuplescited by 1
  11. 2025Plan-and-Refine: Diverse and Comprehensive Retrieval-Augmented Generationcited by 0
  12. 2025SIGIR 2025 -- LiveRAG Challenge Reportcited by 0
  13. 2026From Evidence-Based Medicine to Knowledge Graph: Retrieval-Augmented Generation for Sports Rehabilitation and a Domain Benchmarkcited by 0
  14. 2026Supporting Humans in Evaluating AI Summaries of Legal Depositionscited by 0
  15. 2026A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facilitycited by 0
  16. 2026Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses
  2. 2026Overview of the TREC 2025 Retrieval Augmented Generation (RAG) Track
  3. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions
  4. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  5. 2024ARAGOG: Advanced RAG Output Grading
  6. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  7. 2024Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU
  8. 2024Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems