# Auto Search Indexer for End-to-End Document Retrieval

arXiv: https://arxiv.org/abs/2310.12455 · HTML: https://mlsanity.com/paper/2310.12455/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2023.

## Built on by (most cited first, 16 of 16)

- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation](https://mlsanity.com/paper/2412.11919/) (2024)
- [CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks](https://mlsanity.com/paper/2402.01176/) (2024)
- [A Survey of Generative Search and Recommendation in the Era of Large Language Models](https://mlsanity.com/paper/2404.16924/) (2024)
- [Generative Retrieval and Alignment Model: A New Paradigm for E-commerce Retrieval](https://mlsanity.com/paper/2504.01403/) (2025)
- [Generative Information Retrieval Evaluation](https://mlsanity.com/paper/2404.08137/) (2024)
- [Universal Item Tokenization for Transferable Generative Recommendation](https://mlsanity.com/paper/2504.04405/) (2025)
- [On Synthetic Data Strategies for Domain-Specific Generative Retrieval](https://mlsanity.com/paper/2502.17957/) (2025)
- [Self-Retrieval: End-to-End Information Retrieval with One Large Language Model](https://mlsanity.com/paper/2403.00801/) (2024)
- [Exploring Training and Inference Scaling Laws in Generative Retrieval](https://mlsanity.com/paper/2503.18941/) (2025)
- [Pre-training Generative Recommender with Multi-Identifier Item Tokenization](https://mlsanity.com/paper/2504.04400/) (2025)
- [DiffuGR: Generative Document Retrieval with Diffusion Language Models](https://mlsanity.com/paper/2511.08150/) (2025)
- [It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning](https://mlsanity.com/paper/2609.00638/) (2026)
- [QueStER: Query Specification for Generative keyword-based Retrieval](https://mlsanity.com/paper/2511.05301/) (2025)
- [Chunk Knowledge Generation Model for Enhanced Information Retrieval: A Multi-task Learning Approach](https://mlsanity.com/paper/2509.15658/) (2025)

## Abstract (arXiv)

Generative retrieval, which is a new advanced paradigm for document retrieval, has recently attracted research interests, since it encodes all documents into the model and directly generates the retrieved documents. However, its power is still underutilized since it heavily relies on the "preprocessed" document identifiers (docids), thus limiting its retrieval performance and ability to retrieve new documents. In this paper, we propose a novel fully end-to-end retrieval paradigm. It can not only end-to-end learn the best docids for existing and new documents automatically via a semantic indexing module, but also perform end-to-end document retrieval via an encoder-decoder-based generative model, namely Auto Search Indexer (ASI). Besides, we design a reparameterization mechanism to combine the above two modules into a joint optimization framework. Extensive experimental results demonstrate the superiority of our model over advanced baselines on both public and industrial datasets and also verify the ability to deal with new documents.