# Enhancing Embedding Representation Stability in Recommendation Systems with Semantic ID

arXiv: https://arxiv.org/abs/2504.02137 · HTML: https://mlsanity.com/paper/2504.02137/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [A Survey on Learning to Hash](https://mlsanity.com/paper/1606.00185/) (2016)
- [Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders](https://mlsanity.com/paper/2210.12316/) (2022)
- [Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations](https://mlsanity.com/paper/2306.08121/) (2023)
- [Model Size Reduction Using Frequency Based Double Hashing for Recommender Systems](https://mlsanity.com/paper/2007.14523/) (2020)
- [Monolith: Real Time Recommendation System With Collisionless Embedding Table](https://mlsanity.com/paper/2209.07663/) (2022)
- [On the Dimensionality of Embeddings for Sparse Features and Data](https://mlsanity.com/paper/1901.02103/) (2019)

## Built on by (most cited first, 19 of 19)

- [PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations](https://mlsanity.com/paper/2510.07784/) (2025)
- [RecoWorld: Building Simulated Environments for Agentic Recommender Systems](https://mlsanity.com/paper/2509.10397/) (2025)
- [MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation](https://mlsanity.com/paper/2508.15281/) (2025)
- [Understanding Generative Recommendation with Semantic IDs from a Model-scaling View](https://mlsanity.com/paper/2509.25522/) (2025)
- [NEZHA: A Zero-sacrifice and Hyperspeed Decoding Architecture for Generative Recommendations](https://mlsanity.com/paper/2511.18793/) (2025)
- [Stop Treating Collisions Equally: Qualification-Aware Semantic ID Learning for Recommendation at Industrial Scale](https://mlsanity.com/paper/2603.00632/) (2026)
- [LLM-Aligned Geographic Item Tokenization for Local-Life Recommendation](https://mlsanity.com/paper/2511.14221/) (2025)
- [When Text-as-Vision Meets Semantic IDs in Generative Recommendation: An Empirical Study](https://mlsanity.com/paper/2601.14697/) (2026)
- [PCR-CA: Parallel Codebook Representations with Contrastive Alignment for Multiple-Category App Recommendation](https://mlsanity.com/paper/2508.18166/) (2025)
- [LLMs Need Encoders for Semantic IDs Too](https://mlsanity.com/paper/2606.00324/) (2026)
- [DCN^2: Interplay of Implicit Collision Weights and Explicit Cross Layers for Large-Scale Recommendation](https://mlsanity.com/paper/2506.21624/) (2025)
- [DMGIN: How Multimodal LLMs Enhance Large Recommendation Models for Lifelong User Post-click Behaviors](https://mlsanity.com/paper/2508.21801/) (2025)
- [The Best of the Two Worlds: Harmonizing Semantic and Hash IDs for Sequential Recommendation](https://mlsanity.com/paper/2512.10388/) (2025)
- [FLUID: From Ephemeral IDs to Multimodal Semantic Codes for Industrial-Scale Livestreaming Recommendation](https://mlsanity.com/paper/2605.21832/) (2026)
- [LLM Retrieval for Stable and Predictable Ad Recommendations](https://mlsanity.com/paper/2605.21969/) (2026)
- [OneShot: Index-in-Ranking with Neural Scoring for Large-Scale Retrieval](https://mlsanity.com/paper/2607.27475/) (2026)
- [Interpretable Representation via LLM-Driven Generative Disentanglement for Local-Life Service Recommendation](https://mlsanity.com/paper/2607.27944/) (2026)
- [One Hierarchy, Two Systems: Semantic Product IDs for Discovery-Surface Ranking and Search-Page Query Reformulation](https://mlsanity.com/paper/2608.20640/) (2026)
- [Preference Shapes Relevance: Cross-component Hierarchical Semantic Alignment for Personalized Generative Retrieval](https://mlsanity.com/paper/2608.30553/) (2026)

## Abstract (arXiv)

The exponential growth of online content has posed significant challenges to ID-based models in industrial recommendation systems, ranging from extremely high cardinality and dynamically growing ID space, to highly skewed engagement distributions, to prediction instability as a result of natural id life cycles (e.g, the birth of new IDs and retirement of old IDs). To address these issues, many systems rely on random hashing to handle the id space and control the corresponding model parameters (i.e embedding table). However, this approach introduces data pollution from multiple ids sharing the same embedding, leading to degraded model performance and embedding representation instability. This paper examines these challenges and introduces Semantic ID prefix ngram, a novel token parameterization technique that significantly improves the performance of the original Semantic ID. Semantic ID prefix ngram creates semantically meaningful collisions by hierarchically clustering items based on their content embeddings, as opposed to random assignments. Through extensive experimentation, we demonstrate that Semantic ID prefix ngram not only addresses embedding instability but also significantly improves tail id modeling, reduces overfitting, and mitigates representation shifts. We further highlight the advantages of Semantic ID prefix ngram in attention-based models that contextualize user histories, showing substantial performance improvements. We also report our experience of integrating Semantic ID into Meta production Ads Ranking system, leading to notable performance gains and enhanced prediction stability in live deployments.