# Trie-Aware Transformers for Generative Recommendation

arXiv: https://arxiv.org/abs/2602.21677 · HTML: https://mlsanity.com/paper/2602.21677/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 49 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022)
- [Session-based Recommendations with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06939/) (2015)
- [Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding](https://mlsanity.com/paper/1809.07426/) (2018)
- [Text Is All You Need: Learning Language Representations for Sequential Recommendation](https://mlsanity.com/paper/2305.13731/) (2023)
- [Learnable Item Tokenization for Generative Recommendation](https://mlsanity.com/paper/2405.07314/) (2024)
- [From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviews](https://mlsanity.com/paper/1303.4402/) (2013)
- [Learning Tree-based Deep Model for Recommender Systems](https://mlsanity.com/paper/1801.02294/) (2018)
- [Generative Recommendation: Towards Next-generation Recommender Paradigm](https://mlsanity.com/paper/2304.03516/) (2023)
- [Bootstrap Latent Representations for Multi-modal Recommendation](https://mlsanity.com/paper/2207.05969/) (2022)
- [Joint Optimization of Tree-based Index and Deep Model for Recommender Systems](https://mlsanity.com/paper/1902.07565/) (2019)
- [Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations](https://mlsanity.com/paper/2503.02453/) (2025)
- [Tree-CNN: A Hierarchical Deep Convolutional Neural Network for Incremental Learning](https://mlsanity.com/paper/1802.05800/) (2018)
- [Learning Optimal Tree Models Under Beam Search](https://mlsanity.com/paper/2006.15408/) (2020)
- [CoST: Contrastive Quantization based Semantic Tokenization for Generative Recommendation](https://mlsanity.com/paper/2404.14774/) (2024)
- [Personalized Behavior-Aware Transformer for Multi-Behavior Sequential Recommendation](https://mlsanity.com/paper/2402.14473/) (2024)
- [LLMTreeRec: Unleashing the Power of Large Language Models for Cold-Start Recommendations](https://mlsanity.com/paper/2404.00702/) (2024)
- [HiD-VAE: Interpretable Generative Recommendation via Hierarchical and Disentangled Semantic IDs](https://mlsanity.com/paper/2508.04618/) (2025)
- [CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendation](https://mlsanity.com/paper/2511.22707/) (2025)
- [MSL: Not All Tokens Are What You Need for Tuning LLM as a Recommender](https://mlsanity.com/paper/2504.04178/) (2025)
- [DiscRec: Disentangled Semantic-Collaborative Modeling for Generative Recommendation](https://mlsanity.com/paper/2506.15576/) (2025)
- [UNGER: Generative Recommendation with A Unified Code via Semantic and Collaborative Integration](https://mlsanity.com/paper/2502.06269/) (2025)
- [A Simple Contrastive Framework Of Item Tokenization For Generative Recommendation](https://mlsanity.com/paper/2506.16683/) (2025)
- [Act-With-Think: Chunk Auto-Regressive Modeling for Generative Recommendation](https://mlsanity.com/paper/2506.23643/) (2025)

## Abstract (arXiv)

Generative recommendation (GR) aligns with advances in generative AI by casting next-item prediction as token-level generation rather than score-based ranking. Most GR methods adopt a two-stage pipeline: (i) textititem tokenization, which maps each item to a sequence of discrete, hierarchically organized tokens; and (ii) textitautoregressive generation, which predicts the next item's tokens conditioned on the tokens of user's interaction history. Although hierarchical tokenization induces a prefix tree (trie) over items, standard autoregressive modeling with conventional Transformers often flattens item tokens into a linear stream and overlooks the underlying topology. To address this, we propose TrieRec, a trie-aware generative recommendation method that augments Transformers with structural inductive biases via two positional encodings. First, a textittrie-aware absolute positional encoding aggregates a token's (node's) local structural context (eg depth, ancestors, and descendants) into the token representation. Second, a textittopology-aware relative positional encoding injects pairwise structural relations into self-attention to capture topology-induced semantic relatedness. TrieRec is also model-agnostic, efficient, and hyperparameter-free. In our experiments, we implement TrieRec within three representative GR backbones, achieving notably improvements of 8.83% on average across four real-world datasets.