# Generative Retrieval as Multi-Vector Dense Retrieval arXiv: https://arxiv.org/abs/2404.00684 ยท HTML: https://mlsanity.com/paper/2404.00684/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 54 references that are in the map. - First posted on arXiv 31 Mar 2024. ## Builds on (most cited first, 30 of 32) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020) - [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016) - [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020) - [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022) - [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021) - [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021) - [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021) - [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://mlsanity.com/paper/2107.05720/) (2021) - [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022) - [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020) - [COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List](https://mlsanity.com/paper/2104.07186/) (2021) - [Autoregressive Search Engines: Generating Substrings as Document Identifiers](https://mlsanity.com/paper/2204.10628/) (2022) - [A Neural Corpus Indexer for Document Retrieval](https://mlsanity.com/paper/2206.02743/) (2022) - [Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation](https://mlsanity.com/paper/2206.10128/) (2022) - [How Does Generative Retrieval Scale to Millions of Passages?](https://mlsanity.com/paper/2305.11841/) (2023) - [Learning to Rank in Generative Retrieval](https://mlsanity.com/paper/2306.15222/) (2023) - [Scalable and Effective Generative Information Retrieval](https://mlsanity.com/paper/2311.09134/) (2023) - [Accurate Word Alignment Induction from Neural Machine Translation](https://mlsanity.com/paper/2004.14837/) (2020) - [Rethinking the Role of Token Retrieval in Multi-Vector Retrieval](https://mlsanity.com/paper/2304.01982/) (2023) - [Cross-lingual Information Retrieval with BERT](https://mlsanity.com/paper/2004.13005/) (2020) - [Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies](https://mlsanity.com/paper/2305.15115/) (2023) - [Continual Learning for Generative Retrieval over Dynamic Corpora](https://mlsanity.com/paper/2308.14968/) (2023) - [CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval](https://mlsanity.com/paper/2211.10411/) (2022) - [GLEN: Generative Retrieval via Lexical Index Learning](https://mlsanity.com/paper/2311.03057/) (2023) - [A White Box Analysis of ColBERT](https://mlsanity.com/paper/2012.09650/) (2020) - [SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes](https://mlsanity.com/paper/2302.06587/) (2023) - [Generative Dense Retrieval: Memory Can Be a Burden](https://mlsanity.com/paper/2401.10487/) (2024) ## Built on by (most cited first, 8 of 8) - [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024) - [DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning](https://mlsanity.com/paper/2509.23883/) (2025) - [Balancing the Blend: An Experimental Analysis of Trade-offs in Hybrid Search](https://mlsanity.com/paper/2508.01405/) (2025) - [Purely Semantic Indexing for LLM-based Generative Recommendation and Retrieval](https://mlsanity.com/paper/2509.16446/) (2025) - [Efficient Generative Retrieval for E-commerce Search with Semantic Cluster IDs and Expert-Guided RL](https://mlsanity.com/paper/2605.14434/) (2026) - [Foundations of GenIR](https://mlsanity.com/paper/2501.02842/) (2025) - [Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusion](https://mlsanity.com/paper/2603.06397/) (2026) - [QChunker: Learning Question-Aware Text Chunking for Domain RAG via Multi-Agent Debate](https://mlsanity.com/paper/2603.11650/) (2026) ## Abstract (arXiv) Generative retrieval generates identifiers of relevant documents in an end-to-end manner using a sequence-to-sequence architecture for a given query. The relation between generative retrieval and other retrieval methods, especially those based on matching within dense retrieval models, is not yet fully comprehended. Prior work has demonstrated that generative retrieval with atomic identifiers is equivalent to single-vector dense retrieval. Accordingly, generative retrieval exhibits behavior analogous to hierarchical search within a tree index in dense retrieval when using hierarchical semantic identifiers. However, prior work focuses solely on the retrieval stage without considering the deep interactions within the decoder of generative retrieval. In this paper, we fill this gap by demonstrating that generative retrieval and multi-vector dense retrieval share the same framework for measuring the relevance to a query of a document. Specifically, we examine the attention layer and prediction head of generative retrieval, revealing that generative retrieval can be understood as a special case of multi-vector dense retrieval. Both methods compute relevance as a sum of products of query and document vectors and an alignment matrix. We then explore how generative retrieval applies this framework, employing distinct strategies for computing document token vectors and the alignment matrix. We have conducted experiments to verify our conclusions and show that both paradigms exhibit commonalities of term matching in their alignment matrix.