mlsanityData to 14 Oct 2026

The Viability of Crowdsourcing for RAG Evaluation

arXiv 2504.15689 ↗First posted 22 Apr 2025cs.IRcited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

How good are humans at writing and judging responses in retrieval-augmented generation (RAG) scenarios? To answer this question, we investigate the efficacy of crowdsourcing for RAG through two complementary studies: response writing and response utility judgment. We present the Crowd RAG Corpus 2025 (CrowdRAG-25), which consists of 903 human-written and 903 LLM-generated responses for the 301 topics of the TREC RAG'24 track, across the three discourse styles 'bulleted list', 'essay', and 'news'. For a selection of 65 topics, the corpus further contains 47,320 pairwise human judgments and 10,556 pairwise LLM judgments across seven utility dimensions (e.g., coverage and coherence). Our analyses give insights into human writing behavior for RAG and the viability of crowdsourcing for RAG evaluation. Human pairwise judgments provide reliable and cost-effective results compared to LLM-based pairwise or human/LLM-based pointwise judgments, as well as automated comparisons with human-written reference responses. All our data and tools are freely available.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 32 of its 53 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

32 of 53 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework14 Nov
  2. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems25 Jun
  3. 2024Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track24 Jun
  4. 2024DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation9 Jun
  5. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  7. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 25

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  5. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  6. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  7. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  8. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  9. 2023Large language models can accurately predict searcher preferencescited by 131
  10. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 85
  11. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 69
  12. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 56
13 more
  1. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  2. 2023One-Shot Labeling for Automatic Relevance Estimationcited by 31
  3. 2024LLM-based relevance assessment still can't replace human relevance assessmentcited by 27
  4. 2024Don't Use LLMs to Make Relevance Judgmentscited by 26
  5. 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Lookcited by 18
  6. 2024Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metricscited by 17
  7. 2024LLMs Can Patch Up Missing Relevance Judgments in Evaluationcited by 12
  8. 2023Evaluating Generative Ad Hoc Information Retrievalcited by 11
  9. 2024A Workbench for Autograding Retrieve/Generate Systemscited by 9
  10. 2024SynDL: A Large-Scale Synthetic Test Collection for Passage Retrievalcited by 9
  11. 2024JudgeBlender: Ensembling Judgments for Automatic Relevance Assessmentcited by 6
  12. 2024Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrievalcited by 2
  13. 2025GenTREC: The First Test Collection Generated by Large Language Models for Evaluating Information Retrieval Systemscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Towards Transparent RAG: Fostering Evidence Traceability in LLM Generation via Reinforcement Learningcited by 4
  2. 2026AHEAD: Advancing Multi-Class Label Aggregation with Interpretable Cross-Annotator Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Human-Centered Evaluation of RAG outputs: a framework and questionnaire for human-AI collaboration
  2. 2025GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation
  3. 2026The RAT: A Unified Bayesian Model for RAG Evaluation
  4. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
  5. 2025Evaluating the Effect of Retrieval Augmentation on Social Biases
  6. 2023Ragas: Automated Evaluation of Retrieval Augmented Generation
  7. 2025Knowledge-Graph Based RAG System Evaluation Framework
  8. 2025Towards a rigorous evaluation of RAG systems: the challenge of due diligence