# Understanding Differential Search Index for Text Retrieval

arXiv: https://arxiv.org/abs/2305.02073 · HTML: https://mlsanity.com/paper/2305.02073/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2023.

## Built on by (most cited first, 9 of 9)

- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [How Does Generative Retrieval Scale to Millions of Passages?](https://mlsanity.com/paper/2305.11841/) (2023)
- [DBCopilot: Natural Language Querying over Massive Databases via Schema Routing](https://mlsanity.com/paper/2312.03463/) (2023)
- [Reindex-Then-Adapt: Improving Large Language Models for Conversational Recommendation](https://mlsanity.com/paper/2405.12119/) (2024)
- [Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search](https://mlsanity.com/paper/2402.07742/) (2024)
- [On Synthetic Data Strategies for Domain-Specific Generative Retrieval](https://mlsanity.com/paper/2502.17957/) (2025)
- [A Simple Contrastive Framework Of Item Tokenization For Generative Recommendation](https://mlsanity.com/paper/2506.16683/) (2025)
- [Event GDR: Event-Centric Generative Document Retrieval](https://mlsanity.com/paper/2405.06886/) (2024)
- [Generative Universal Multimodal Retrieval with Dual-role Identifiers](https://mlsanity.com/paper/2608.12987/) (2026)

## Abstract (arXiv)

The Differentiable Search Index (DSI) is a novel information retrieval (IR) framework that utilizes a differentiable function to generate a sorted list of document identifiers in response to a given query. However, due to the black-box nature of the end-to-end neural architecture, it remains to be understood to what extent DSI possesses the basic indexing and retrieval abilities. To mitigate this gap, in this study, we define and examine three important abilities that a functioning IR framework should possess, namely, exclusivity, completeness, and relevance ordering. Our analytical experimentation shows that while DSI demonstrates proficiency in memorizing the unidirectional mapping from pseudo queries to document identifiers, it falls short in distinguishing relevant documents from random ones, thereby negatively impacting its retrieval effectiveness. To address this issue, we propose a multi-task distillation approach to enhance the retrieval quality without altering the structure of the model and successfully endow it with improved indexing abilities. Through experiments conducted on various datasets, we demonstrate that our proposed method outperforms previous DSI baselines.