Resources for Automated Evaluation of Assistive RAG Systems that Help Readers with News Trustworthiness Assessment
On the map: Retrieval-augmented generation → RAG pipeline design and evaluation
Abstract (arXiv)
Many readers today struggle to assess the trustworthiness of online news because reliable reporting coexists with misinformation. The TREC 2025 DRAGUN (Detection, Retrieval, and Augmented Generation for Understanding News) Track provided a venue for researchers to develop and evaluate assistive RAG systems that support readers' news trustworthiness assessment by producing reader-oriented, well-attributed reports. As the organizers of the DRAGUN track, we describe the resources that we have newly developed to allow for the reuse of the track's tasks. The track had two tasks: (Task 1) Question Generation, producing 10 ranked investigative questions; and (Task 2, the main task) Report Generation, producing a 250-word report grounded in the MS MARCO V2.1 Segmented Corpus. As part of the track's evaluation, we had TREC assessors create importance-weighted rubrics of questions with expected short answers for 30 different news articles. These rubrics represent the information that assessors believe is important for readers to assess an article's trustworthiness. The assessors then used their rubrics to manually judge the participating teams' submitted runs. To make these tasks and their rubrics reusable, we have created an automated process to judge runs not part of the original assessing. We show that our AutoJudge ranks existing runs well compared to the TREC human-assessed evaluation (Kendall's τ = 0.678 for Task 1 and τ = 0.872 for Task 2). These resources enable both the evaluation of RAG systems for assistive news trustworthiness assessment and, with the human evaluation as a benchmark, research on improving automated RAG evaluation.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 11 of its 35 references that are in the map. First posted on arXiv 27 Feb 2026.
Builds on
11 of 35 references.
Closest ancestors · 1
Same direction, newest first
- 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models21 Apr
Other references · 10
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
- 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
- 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
- 2021A Survey on Automated Fact-Checkingcited by 376
- 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 361
- 2021Automated Fact-Checking for Assisting Human Fact-Checkerscited by 144
- 2023The science of fake newscited by 94
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026From Questions to Trust Reports: A LLM-IR Framework for the TREC 2025 DRAGUN Track ↗
- 2024Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework
- 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
- 2026Overview of the TREC 2025 Retrieval Augmented Generation (RAG) Track
- 2025ClaimTrust: Propagation Trust Scoring for RAG Systems
- 2025Auto-ARGUE: LLM-Based Report Generation Evaluation
- 2026Assessing the Downstream Utility of Evidence-Aware Retrieval in RAG
- 2025The Viability of Crowdsourcing for RAG Evaluation