# How Does Generative Retrieval Scale to Millions of Passages?

arXiv: https://arxiv.org/abs/2305.11841 · HTML: https://mlsanity.com/paper/2305.11841/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 51 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [Recommender Systems with Generative Retrieval](https://mlsanity.com/paper/2305.05065/) (2023)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022)
- [XOR QA: Cross-lingual Open-Retrieval Question Answering](https://mlsanity.com/paper/2010.11856/) (2020)
- [RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses](https://mlsanity.com/paper/2210.10634/) (2022)
- [Autoregressive Search Engines: Generating Substrings as Document Identifiers](https://mlsanity.com/paper/2204.10628/) (2022)
- [A Neural Corpus Indexer for Document Retrieval](https://mlsanity.com/paper/2206.02743/) (2022)
- [The Efficiency Misnomer](https://mlsanity.com/paper/2110.12894/) (2021)
- [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://mlsanity.com/paper/2101.05667/) (2021)
- [Learning to Tokenize for Generative Retrieval](https://mlsanity.com/paper/2304.04171/) (2023)
- [Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation](https://mlsanity.com/paper/2206.10128/) (2022)
- [End-to-End Retrieval in Continuous Space](https://mlsanity.com/paper/1811.08008/) (2018)
- [DSI++: Updating Transformer Memory with New Documents](https://mlsanity.com/paper/2212.09744/) (2022)

## Built on by (most cited first, 30 of 48)

- [LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://mlsanity.com/paper/2306.15626/) (2023)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models](https://mlsanity.com/paper/2309.15088/) (2023)
- [Learning to Rank in Generative Retrieval](https://mlsanity.com/paper/2306.15222/) (2023)
- [Source-Aware Training Enables Knowledge Attribution in Language Models](https://mlsanity.com/paper/2404.01019/) (2024)
- [FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering](https://mlsanity.com/paper/2508.14052/) (2025)
- [Language Models As Semantic Indexers](https://mlsanity.com/paper/2310.07815/) (2023)
- [Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding](https://mlsanity.com/paper/2404.14600/) (2024)
- [Large Search Model: Redefining Search Stack in the Era of LLMs](https://mlsanity.com/paper/2310.14587/) (2023)
- [Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models](https://mlsanity.com/paper/2402.07179/) (2024)
- [GLEN: Generative Retrieval via Lexical Index Learning](https://mlsanity.com/paper/2311.03057/) (2023)
- [A Survey of Generative Search and Recommendation in the Era of Large Language Models](https://mlsanity.com/paper/2404.16924/) (2024)
- [Learning Facts at Scale with Active Reading](https://mlsanity.com/paper/2508.09494/) (2025)
- [Dense Passage Retrieval: Is it Retrieving?](https://mlsanity.com/paper/2402.11035/) (2024)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [A Generative Approach for Wikipedia-Scale Visual Entity Recognition](https://mlsanity.com/paper/2403.02041/) (2024)
- [Beyond Memorization: The Challenge of Random Memory Access in Language Models](https://mlsanity.com/paper/2403.07805/) (2024)
- [Generative Retrieval as Multi-Vector Dense Retrieval](https://mlsanity.com/paper/2404.00684/) (2024)
- [Generative Information Retrieval Evaluation](https://mlsanity.com/paper/2404.08137/) (2024)
- [Exploring the Practicality of Generative Retrieval on Dynamic Corpora](https://mlsanity.com/paper/2305.18952/) (2023)
- [From RAG to RICHES: Retrieval Interlaced with Sequence Generation](https://mlsanity.com/paper/2407.00361/) (2024)
- [Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach](https://mlsanity.com/paper/2410.23676/) (2024)
- [On Synthetic Data Strategies for Domain-Specific Generative Retrieval](https://mlsanity.com/paper/2502.17957/) (2025)
- [Self-Retrieval: End-to-End Information Retrieval with One Large Language Model](https://mlsanity.com/paper/2403.00801/) (2024)
- [One size doesn't fit all: Predicting the Number of Examples for In-Context Learning](https://mlsanity.com/paper/2403.06402/) (2024)
- [A Survey of Generative Information Retrieval](https://mlsanity.com/paper/2406.01197/) (2024)
- [Exploring Training and Inference Scaling Laws in Generative Retrieval](https://mlsanity.com/paper/2503.18941/) (2025)
- [MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora](https://mlsanity.com/paper/2507.09924/) (2025)
- [CTRL-Rec: Controlling Recommender Systems With Natural Language](https://mlsanity.com/paper/2510.12742/) (2025)

## Abstract (arXiv)

Popularized by the Differentiable Search Index, the emerging paradigm of generative retrieval re-frames the classic information retrieval problem into a sequence-to-sequence modeling task, forgoing external indices and encoding an entire document corpus within a single Transformer. Although many different approaches have been proposed to improve the effectiveness of generative retrieval, they have only been evaluated on document corpora on the order of 100k in size. We conduct the first empirical study of generative retrieval techniques across various corpus scales, ultimately scaling up to the entire MS MARCO passage ranking task with a corpus of 8.8M passages and evaluating model sizes up to 11B parameters. We uncover several findings about scaling generative retrieval to millions of passages; notably, the central importance of using synthetic queries as document representations during indexing, the ineffectiveness of existing proposed architecture modifications when accounting for compute cost, and the limits of naively scaling model parameters with respect to retrieval performance. While we find that generative retrieval is competitive with state-of-the-art dual encoders on small corpora, scaling to millions of passages remains an important and unsolved challenge. We believe these findings will be valuable for the community to clarify the current state of generative retrieval, highlight the unique challenges, and inspire new research directions.