# Generative Information Retrieval Evaluation

arXiv: https://arxiv.org/abs/2404.08137 · HTML: https://mlsanity.com/paper/2404.08137/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 17 of its 82 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 17 of 17)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models](https://mlsanity.com/paper/2401.01313/) (2024)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Learning to Tokenize for Generative Retrieval](https://mlsanity.com/paper/2304.04171/) (2023)
- [How Does Generative Retrieval Scale to Millions of Passages?](https://mlsanity.com/paper/2305.11841/) (2023)
- [Evaluating the Robustness of Retrieval Pipelines with Query Variation Generators](https://mlsanity.com/paper/2111.13057/) (2021)
- [User Simulation for Evaluating Information Access Systems](https://mlsanity.com/paper/2306.08550/) (2023)
- [Continual Learning for Generative Retrieval over Dynamic Corpora](https://mlsanity.com/paper/2308.14968/) (2023)
- [A Unified Generative Retriever for Knowledge-Intensive Language Tasks via Prompt Learning](https://mlsanity.com/paper/2304.14856/) (2023)
- [Auto Search Indexer for End-to-End Document Retrieval](https://mlsanity.com/paper/2310.12455/) (2023)
- [Predicting Efficiency/Effectiveness Trade-offs for Dense vs. Sparse Retrieval Strategy Selection](https://mlsanity.com/paper/2109.10739/) (2021)
- [Generating Multi-Aspect Queries for Conversational Search](https://mlsanity.com/paper/2403.19302/) (2024)
- [The Infinite Index: Information Retrieval on Generative Text-To-Image Models](https://mlsanity.com/paper/2212.07476/) (2022)
- [Improving Content Retrievability in Search with Controllable Query Generation](https://mlsanity.com/paper/2303.11648/) (2023)
- [Assessing Efficiency-Effectiveness Tradeoffs in Multi-Stage Retrieval Systems Without Using Relevance Judgments](https://mlsanity.com/paper/1506.00717/) (2015)
- [The Evolution of Web Search User Interfaces -- An Archaeological Analysis of Google Search Engine Result Pages](https://mlsanity.com/paper/2301.08613/) (2023)
- [Context-Driven Interactive Query Simulations Based on Generative Large Language Models](https://mlsanity.com/paper/2312.09631/) (2023)

## Built on by (most cited first, 7 of 7)

- [Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track](https://mlsanity.com/paper/2406.16828/) (2024)
- [Auto-ARGUE: LLM-Based Report Generation Evaluation](https://mlsanity.com/paper/2509.26184/) (2025)
- [exHarmony: Authorship and Citations for Benchmarking the Reviewer Assignment Problem](https://mlsanity.com/paper/2502.07683/) (2025)
- [Measuring text summarization factuality using atomic facts entailment metrics in the context of retrieval augmented generation](https://mlsanity.com/paper/2408.15171/) (2024)
- [Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?](https://mlsanity.com/paper/2601.13227/) (2026)
- [Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2605.14488/) (2026)
- [Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning](https://mlsanity.com/paper/2610.01936/) (2026)

## Abstract (arXiv)

In this chapter, we consider generative information retrieval evaluation from two distinct but interrelated perspectives. First, large language models (LLMs) themselves are rapidly becoming tools for evaluation, with current research indicating that LLMs may be superior to crowdsource workers and other paid assessors on basic relevance judgement tasks. We review past and ongoing related research, including speculation on the future of shared task initiatives, such as TREC, and a discussion on the continuing need for human assessments. Second, we consider the evaluation of emerging LLM-based generative information retrieval (GenIR) systems, including retrieval augmented generation (RAG) systems. We consider approaches that focus both on the end-to-end evaluation of GenIR systems and on the evaluation of a retrieval component as an element in a RAG system. Going forward, we expect the evaluation of GenIR systems to be at least partially based on LLM-based assessment, creating an apparent circularity, with a system seemingly evaluating its own output. We resolve this apparent circularity in two ways: 1) by viewing LLM-based assessment as a form of "slow search", where a slower IR system is used for evaluation and training of a faster production IR system; and 2) by recognizing a continuing need to ground evaluation in human assessment, even if the characteristics of that human assessment must change.