# Auto-ARGUE: LLM-Based Report Generation Evaluation

arXiv: https://arxiv.org/abs/2509.26184 · HTML: https://mlsanity.com/paper/2509.26184/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 10 of its 17 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 10 of 10)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework](https://mlsanity.com/paper/2411.09607/) (2024)
- [The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models](https://mlsanity.com/paper/2504.15068/) (2025)
- [On the Evaluation of Machine-Generated Reports](https://mlsanity.com/paper/2405.00982/) (2024)
- [GINGER: Grounded Information Nugget-Based Generation of Responses](https://mlsanity.com/paper/2503.18174/) (2025)
- [Overview of the TREC 2024 NeuCLIR Track](https://mlsanity.com/paper/2509.14355/) (2025)
- [Generative Information Retrieval Evaluation](https://mlsanity.com/paper/2404.08137/) (2024)

## Built on by (most cited first, 14 of 14)

- [Principled Context Engineering for RAG: Statistical Guarantees via Conformal Prediction](https://mlsanity.com/paper/2511.17908/) (2025)
- [Query Decomposition for RAG: Balancing Exploration-Exploitation](https://mlsanity.com/paper/2510.18633/) (2025)
- [Overview of the TREC 2025 RAGTIME Track](https://mlsanity.com/paper/2602.10024/) (2026)
- [Incorporating Q&A Nuggets into Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.13222/) (2026)
- [LANCER: LLM Reranking for Nugget Coverage](https://mlsanity.com/paper/2601.22008/) (2026)
- [CoverageBench: Evaluating Information Coverage across Tasks and Domains](https://mlsanity.com/paper/2603.20034/) (2026)
- [Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2510.24870/) (2025)
- [Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?](https://mlsanity.com/paper/2601.13227/) (2026)
- [Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage](https://mlsanity.com/paper/2603.08819/) (2026)
- [DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation](https://mlsanity.com/paper/2605.04458/) (2026)
- [Supporting Humans in Evaluating AI Summaries of Legal Depositions](https://mlsanity.com/paper/2601.15182/) (2026)
- [Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization](https://mlsanity.com/paper/2604.21072/) (2026)
- [Retrieval, not hallucinations, will be the limiting factor for LLM-based clinical AI tools](https://mlsanity.com/paper/2607.24793/) (2026)
- [Knowledge Pull Requests for Continual Document Authoring](https://mlsanity.com/paper/2609.26634/) (2026)

## Abstract (arXiv)

Generation of citation-backed reports is a primary use case for retrieval-augmented generation (RAG) systems. While open-source evaluation tools exist for various RAG tasks, tools designed for report generation are lacking. Accordingly, we introduce Auto-ARGUE, a robust LLM-based implementation of the recently proposed ARGUE framework for report generation evaluation. We present analysis of Auto-ARGUE on the report generation pilot task from the TREC 2024 NeuCLIR track and on two tasks from the TREC 2024 RAG track, showing good system-level correlations with human judgments. Additionally, we release ARGUE-Viz, a web app for visualization and fine-grained analysis of Auto-ARGUE judgments and scores.