# TokenMinds: Pretrained User Tokens and Embeddings for User Understanding in Large Recommender Systems

arXiv: https://arxiv.org/abs/2606.25147 · HTML: https://mlsanity.com/paper/2606.25147/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [On the Theoretical Limitations of Embedding-Based Retrieval](https://mlsanity.com/paper/2508.21038/) (2025)
- [PALR: Personalization Aware LLMs for Recommendation](https://mlsanity.com/paper/2305.07622/) (2023)
- [Pathways: Asynchronous Distributed Dataflow for ML](https://mlsanity.com/paper/2203.12533/) (2022)
- [IDGenRec: LLM-RecSys Alignment with Textual ID Learning](https://mlsanity.com/paper/2403.19021/) (2024)
- [OneRec-Think: In-Text Reasoning for Generative Recommendation](https://mlsanity.com/paper/2510.11639/) (2025)
- [LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders](https://mlsanity.com/paper/2505.04421/) (2025)
- [Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations](https://mlsanity.com/paper/2503.02453/) (2025)
- [Mixed Information Flow for Cross-domain Sequential Recommendations](https://mlsanity.com/paper/2012.00485/) (2020)
- [LLM4MSR: An LLM-Enhanced Paradigm for Multi-Scenario Recommendation](https://mlsanity.com/paper/2406.12529/) (2024)
- [Unified Embedding: Battle-Tested Feature Representations for Web-Scale ML Systems](https://mlsanity.com/paper/2305.12102/) (2023)
- [GPR: Towards a Generative Pre-trained One-Model Paradigm for Large-Scale Advertising Recommendation](https://mlsanity.com/paper/2511.10138/) (2025)
- [DV365: Extremely Long User History Modeling at Instagram](https://mlsanity.com/paper/2506.00450/) (2025)
- [MLoRA: Multi-Domain Low-Rank Adaptive Network for CTR Prediction](https://mlsanity.com/paper/2408.08913/) (2024)
- [Empowering General-purpose User Representation with Full-life Cycle Behavior Modeling](https://mlsanity.com/paper/2110.11337/) (2021)
- [Latent User Intent Modeling for Sequential Recommenders](https://mlsanity.com/paper/2211.09832/) (2022)

## Built on by (most cited first, 1 of 1)

- [Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems](https://mlsanity.com/paper/2609.30001/) (2026)

## Abstract (arXiv)

User modeling in industrial recommender systems typically produces dense embeddings, which suffer from representational constraints inherent to fixed-dimensional vectors. An emerging alternative for discrete user representation -- using LLMs to generate text-based user tokens -- captures topical co-occurrences rather than deep sequential behavior dynamics and produces outputs that are difficult to ground to item attributes. Meanwhile, Semantic ID (SID) based item tokenization has proven effective for improving generalization in generative recommendation, yet discrete SID-based representations for users remain largely unexplored. We propose TokenMinds, an industrial-scale system that extends the PLUM framework from item retrieval to user modeling, generating both discrete SID-based user tokens and dense user embeddings via an encoder-decoder architecture adapted from pre-trained LLMs. This dual-output design provides the complementary benefits of discrete, semantically grounded user representations while maintaining compatibility with existing downstream models that rely on dense embeddings. Additionally, the shared SID vocabulary naturally extends to cross-scenario modeling: by unifying long-form and short-form video behaviors into a single model, we substantially reduce training and serving costs. We validate TokenMinds through extensive offline experiments and live launches on multiple YouTube surfaces, served on full user traffic (billions of users) via an asynchronous infrastructure that decouples representation generation from downstream scoring. Focusing on ranking as the primary downstream use case, our results confirm the practical viability of SID-based user tokens at industrial scale and demonstrate that tokens and dense embeddings provide complementary value across different production ranking systems.