# Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation

arXiv: https://arxiv.org/abs/2604.02324 · HTML: https://mlsanity.com/paper/2604.02324/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 24 of 24)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://mlsanity.com/paper/2305.11554/) (2023)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation](https://mlsanity.com/paper/2311.09049/) (2023)
- [OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment](https://mlsanity.com/paper/2502.18965/) (2025)
- [Learnable Item Tokenization for Generative Recommendation](https://mlsanity.com/paper/2405.07314/) (2024)
- [A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)](https://mlsanity.com/paper/2404.00579/) (2024)
- [MTGR: Industrial-Scale Generative Recommendation Framework in Meituan](https://mlsanity.com/paper/2505.18654/) (2025)
- [OneRec Technical Report](https://mlsanity.com/paper/2506.13695/) (2025)
- [VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models](https://mlsanity.com/paper/2503.16980/) (2025)
- [Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse](https://mlsanity.com/paper/2311.05139/) (2023)
- [LinkedOut: Linking World Knowledge Representation Out of Video LLM for Next-Generation Video Recommendation](https://mlsanity.com/paper/2512.16891/) (2025)
- [AdANNS: A Framework for Adaptive Semantic Search](https://mlsanity.com/paper/2305.19435/) (2023)
- [How Well Does Generative Recommendation Generalize?](https://mlsanity.com/paper/2603.19809/) (2026)
- [Boosting Knowledge Graph-based Recommendations through Confidence-Aware Augmentation with Large Language Models](https://mlsanity.com/paper/2502.03715/) (2025)

## Built on by (most cited first, 2 of 2)

- [LLMs Need Encoders for Semantic IDs Too](https://mlsanity.com/paper/2606.00324/) (2026)
- [Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation](https://mlsanity.com/paper/2608.07816/) (2026)

## Abstract (arXiv)

Language models (LMs) are increasingly extended with new learnable vocabulary tokens for domain-specific tasks, such as Semantic-ID tokens in generative recommendation. The standard practice initializes these new tokens as the mean of existing vocabulary embeddings, then relies on supervised fine-tuning to learn their representations. We present a systematic analysis of this strategy: through spectral and geometric diagnostics, we show that mean initialization collapses all new tokens into a degenerate subspace, erasing inter-token distinctions that subsequent fine-tuning struggles to fully recover. These findings suggest that token initialization is a key bottleneck when extending LMs with new vocabularies. Motivated by this diagnosis, we propose the Grounded Token Initialization Hypothesis: linguistically grounding novel tokens in the pretrained embedding space before fine-tuning better enables the model to leverage its general-purpose knowledge for novel-token domains. We operationalize this hypothesis as GTI (Grounded Token Initialization), a lightweight grounding stage that, prior to fine-tuning, maps new tokens to distinct, semantically meaningful locations in the pretrained embedding space using only paired linguistic supervision. Despite its simplicity, GTI outperforms both mean initialization and existing auxiliary-task adaptation methods in the majority of evaluation settings across multiple generative recommendation benchmarks, including industry-scale and public datasets. Further analyses show that grounded embeddings produce richer inter-token structure that persists through fine-tuning, corroborating the hypothesis that initialization quality is a key bottleneck in vocabulary extension.