# A Systematic Study of Semantic ID Spaces for Generative Information Retrieval

arXiv: https://arxiv.org/abs/2610.08732 · HTML: https://mlsanity.com/paper/2610.08732/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 37 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 9 of 9)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment](https://mlsanity.com/paper/2502.18965/) (2025)
- [Learning to Tokenize for Generative Retrieval](https://mlsanity.com/paper/2304.04171/) (2023)
- [Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation](https://mlsanity.com/paper/2206.10128/) (2022)
- [Ultron: An Ultimate Retriever on Corpus with a Model-based Indexer](https://mlsanity.com/paper/2208.09257/) (2022)
- [Deploying Semantic ID-based Generative Retrieval for Large-Scale Podcast Discovery at Spotify](https://mlsanity.com/paper/2603.17540/) (2026)
- [Purely Semantic Indexing for LLM-based Generative Recommendation and Retrieval](https://mlsanity.com/paper/2509.16446/) (2025)
- [What Makes a Good Semantic ID for Generative Recommendation? A Reproducibility Study](https://mlsanity.com/paper/2609.24430/) (2026)
- [PRQ-KMeans: Projection Residual Quantization for Semantic ID Tokenization](https://mlsanity.com/paper/2608.24207/) (2026)

## Abstract (arXiv)

Generative Information Retrieval (GIR) has emerged as a transformative paradigm, shifting document retrieval from a traditional "retrieve-and-rank" workflow to sequence-to-sequence generation, where a model directly predicts document identifiers (DocIDs). While the semantic design of these DocIDs is known to be critical for performance, a fundamental question remains under-explored: what makes a good DocID? Current approaches rely heavily on computationally expensive downstream evaluations, hindering systematic analysis and rapid iteration. In this work, we address this challenge by presenting a comprehensive study on the properties, metrics, and trade-offs that define effective numerical DocIDs. Specifically, our contributions are threefold: First, we propose a unified framework that unifies Product Quantization (PQ) and Residual Quantization (RQ), and their hybrid variants within a single design space. This enables us to systematically study key DocID properties, such as hierarchy versus parallelism, as well as the impact of hyperparameters like DocID length and codebook size. Second, we define a suite of training-free, intrinsic metrics, to quantify DocID quality and evaluate structural fidelity without the overhead of full model training. Through extensive experiments on MS MARCO 300K and NQ320K, we analyze how these structural properties influence retrieval effectiveness.