# Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation

arXiv: https://arxiv.org/abs/2608.07816 · HTML: https://mlsanity.com/paper/2608.07816/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in LLM reasoning for recommendation, one of 17 directions in LLM reasoning.
- Builds on 10 of its 44 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Placed in

- LLM reasoning / [LLM reasoning for recommendation](https://mlsanity.com/lr/llm-reasoning-for-recommendation/)

## Builds on (most cited first, 10 of 10)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking](https://mlsanity.com/paper/2601.04720/) (2026)
- [OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment](https://mlsanity.com/paper/2502.18965/) (2025)
- [OneRec-Think: In-Text Reasoning for Generative Recommendation](https://mlsanity.com/paper/2510.11639/) (2025)
- [OpenOneRec Technical Report](https://mlsanity.com/paper/2512.24762/) (2025)
- [Reasoning over Semantic IDs Enhances Generative Recommendation](https://mlsanity.com/paper/2603.23183/) (2026)
- [A Survey on Generative Recommendation: Data, Model, and Tasks](https://mlsanity.com/paper/2510.27157/) (2025)
- [Implicit Reasoning for Large Language Model-based Generative Recommendation](https://mlsanity.com/paper/2606.14142/) (2026)
- [Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation](https://mlsanity.com/paper/2604.02324/) (2026)
- [LLMs Need Encoders for Semantic IDs Too](https://mlsanity.com/paper/2606.00324/) (2026)

## Abstract (arXiv)

Recent advances in generative recommendation (GR) leverage large language models (LLMs) as recommender backbones, enabling LLMs to directly generate recommendations conditioned on item-interaction histories. In these systems, items are often represented through semantic IDs (SIDs) added to the LLM vocabulary as special tokens. Ideally, SIDs imbue item token representations with semantic priors, thereby improving model generalization. However, standard vocabulary expansion typically initializes these tokens as random Gaussian vectors, discarding the SIDs' underlying continuous geometry and forcing the LLM to relearn token relationships from interaction data. To demonstrate the consequences of this design, we first show that training from this initialization tends to organize SID embeddings around item popularity rather than semantics. We further show that, despite partially reducing the reliance on popularity and improving cold item performance, the computationally expensive process of continual pretraining (CPT) fails to reliably recover the original semantic geometry. To address these findings, we propose a simple, parameter-free intervention that initializes SID token embeddings directly from their corresponding centroids in the semantic embedding space. Requiring only a few lines of code and no additional training or inference overhead, this drop-in approach improves pure-SFT Recall@5 by up to 16%, reaches peak performance with up to 40% fewer SFT steps, and improves cold-item Recall@5 by up to 60%. Moreover, on datasets that benefit from additional CPT, centroid initialization reaches comparable performance while requiring half as many CPT epochs. Together, our findings show that preserving SID geometry, beyond shared-prefix structure, provides a simple and effective semantic prior for LLM-based GR.