# Purely Semantic Indexing for LLM-based Generative Recommendation and Retrieval

arXiv: https://arxiv.org/abs/2509.16446 · HTML: https://mlsanity.com/paper/2509.16446/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 19 Sep 2025.

## Builds on (most cited first, 21 of 21)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation](https://mlsanity.com/paper/2311.09049/) (2023)
- [Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders](https://mlsanity.com/paper/2210.12316/) (2022)
- [A Neural Corpus Indexer for Document Retrieval](https://mlsanity.com/paper/2206.02743/) (2022)
- [How to Index Item IDs for Recommendation Foundation Models](https://mlsanity.com/paper/2305.06569/) (2023)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search](https://mlsanity.com/paper/2206.06588/) (2022)
- [IDGenRec: LLM-RecSys Alignment with Textual ID Learning](https://mlsanity.com/paper/2403.19021/) (2024)
- [How Does Generative Retrieval Scale to Millions of Passages?](https://mlsanity.com/paper/2305.11841/) (2023)
- [Ultron: An Ultimate Retriever on Corpus with a Model-based Indexer](https://mlsanity.com/paper/2208.09257/) (2022)
- [Order-agnostic Identifier for Large Language Model-based Generative Recommendation](https://mlsanity.com/paper/2502.10833/) (2025)
- [Unifying Generative and Dense Retrieval for Sequential Recommendation](https://mlsanity.com/paper/2411.18814/) (2024)
- [Content-Based Collaborative Generation for Recommender Systems](https://mlsanity.com/paper/2403.18480/) (2024)
- [Generative Retrieval as Multi-Vector Dense Retrieval](https://mlsanity.com/paper/2404.00684/) (2024)
- [A Survey of Generative Information Retrieval](https://mlsanity.com/paper/2406.01197/) (2024)
- [Summarization-Based Document IDs for Generative Retrieval with Language Models](https://mlsanity.com/paper/2311.08593/) (2023)

## Built on by (most cited first, 3 of 3)

- [T-RoPE: Time-Aware Rotary Position Embedding for Sequential Recommendation](https://mlsanity.com/paper/2609.30576/) (2026)
- [GRAFT: Graph-Distilled Generative Retrieval for Facet-Aware Scientific Literature Exploration](https://mlsanity.com/paper/2608.22381/) (2026)
- [A Systematic Study of Semantic ID Spaces for Generative Information Retrieval](https://mlsanity.com/paper/2610.08732/) (2026)

## Abstract (arXiv)

Semantic identifiers (IDs) have proven effective in adapting large language models for generative recommendation and retrieval. However, existing methods often suffer from semantic ID conflicts, where semantically similar documents (or items) are assigned identical IDs. A common strategy to avoid conflicts is to append a non-semantic token to distinguish them, which introduces randomness and expands the search space, therefore hurting performance. In this paper, we propose purely semantic indexing to generate unique, semantic-preserving IDs without appending non-semantic tokens. We enable unique ID assignment by relaxing the strict nearest-centroid selection and introduce two model-agnostic algorithms: exhaustive candidate matching (ECM) and recursive residual searching (RRS). Extensive experiments on sequential recommendation, product search, and document retrieval tasks demonstrate that our methods improve both overall and cold-start performance, highlighting the effectiveness of ensuring ID uniqueness.