mlsanityData to 7 Oct 2026

Evaluating Generative Ad Hoc Information Retrieval

arXiv 2311.04694 ↗First posted 8 Nov 2023cs.IR cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval. Yet, the established evaluation methodology for ranking-based ad hoc retrieval is not suited for the reliable and reproducible evaluation of generated responses. To lay a foundation for developing new evaluation methods for generative retrieval systems, we survey the relevant literature from the fields of information retrieval and natural language processing, identify search tasks and system architectures in generative retrieval, develop a new user model, and study its operationalization.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 73 of its 171 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

73 of 171 references.

References · 73

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  9. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  10. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  12. 2023In-Context Retrieval-Augmented Language Modelscited by 694
38 more
  1. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  2. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  3. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  4. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  5. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  6. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  7. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  8. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  9. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  10. 2020Autoregressive Entity Retrievalcited by 333
  11. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  12. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  13. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  14. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  15. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  16. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  17. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  18. 2019Document Expansion by Query Predictioncited by 225
  19. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  20. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  21. 2021Overview of the TREC 2020 deep learning trackcited by 193
  22. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  23. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  24. 2017ParlAI: A Dialog Research Software Platformcited by 139
  25. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  26. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  27. 2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 112
  28. 2022A Neural Corpus Indexer for Document Retrievalcited by 107
  29. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 88
  30. 2023Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labelscited by 86
  31. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 84
  32. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  33. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 70
  34. 2022Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generationcited by 47
  35. 2020Hooks in the Headline: Learning to Generate Headlines with Controlled Stylescited by 46
  36. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  37. 2021Grounding 'Grounding' in NLPcited by 44
  38. 2022GERE: Generative Evidence Retrieval for Fact Verificationcited by 38

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 6.

11 papers.

  1. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  2. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  3. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 30
  4. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generationcited by 19
  5. 2025GINGER: Grounded Information Nugget-Based Generation of Responsescited by 16
  6. 2025Characterizing Web Search in The Age of Generative AIcited by 5
  7. 2025The Viability of Crowdsourcing for RAG Evaluationcited by 2
  8. 2024Neural Corrective Machine Unrankingcited by 1
  9. 2025Human-Centered Evaluation of RAG outputs: a framework and questionnaire for human-AI collaborationcited by 1
  10. 2025Q-STRUM Debate: Query-Driven Contrastive Summarization for Recommendation Comparisoncited by 0
  11. 2025Dagstuhl Perspectives Workshop 24352 -- Conversational Agents: A Framework for Evaluation (CAFE): Manifestocited by 0