# TurboGR: An Accelerated Training System for Large-Scale Generative Recommendation

arXiv: https://arxiv.org/abs/2605.13433 · HTML: https://mlsanity.com/paper/2605.13433/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 14 of 14)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [Generative Recommendation: Towards Next-generation Recommender Paradigm](https://mlsanity.com/paper/2304.03516/) (2023)
- [Personalized News Recommendation: Methods and Challenges](https://mlsanity.com/paper/2106.08934/) (2021)
- [Building a Performance Model for Deep Learning Recommendation Model Training on GPUs](https://mlsanity.com/paper/2201.07821/) (2022)
- [FuXi-α: Scaling Recommendation Model with Feature Interaction Enhanced Transformer](https://mlsanity.com/paper/2502.03036/) (2025)
- [Analyzing User Engagement with TikTok's Short Format Video Recommendations using Data Donations](https://mlsanity.com/paper/2301.04945/) (2023)
- [A Survey of Real-World Recommender Systems: Challenges, Constraints, and Industrial Perspectives](https://mlsanity.com/paper/2509.06002/) (2025)
- [Scenario-Wise Rec: A Multi-Scenario Recommendation Benchmark](https://mlsanity.com/paper/2412.17374/) (2024)
- [Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training](https://mlsanity.com/paper/2508.03854/) (2025)
- [FuXi-γ: Efficient Sequential Recommendation with Exponential-Power Temporal Encoder and Diagonal-Sparse Positional Mechanism](https://mlsanity.com/paper/2512.12740/) (2025)

## Abstract (arXiv)

Generative recommendation (GR) has emerged as a promising paradigm that replaces fragmented, scenario-specific architectures with unified Transformer-based models, exhibiting scaling-law behavior where recommendation quality improves systematically with increased model capacity and training data. However, deploying GR at scale on Ascend NPUs faces fundamental system-level challenges. These challenges are further exacerbated on Ascend NPUs due to the absence of high-performance implementations for jagged operators and the architectural mismatch between irregular sparse primitives and NPU's dense-computation-optimized design. In this paper, we present model, an Ascend-affinity training system for generative recommendation that systematically addresses these bottlenecks through three core innovations: (i) Ascend-affinity jagged acceleration, including fusion operators that eliminate padding redundancy and dynamic load balancing that reduces inter-device imbalance from 47% to 2.4%; (ii) distributed communication optimization, comprising hierarchical sparse parallelism, semi-asynchronous training with proven convergence guarantees, and fine-grained pipeline orchestration that sustains 94% NPU utilization; and (iii) negative sampling optimization via asynchronous offloading, jaggedness-aware FP16 quantization, and intra-batch logit sharing that expand the effective negative space without additional embedding lookups. Evaluated on the KuaiRand-27K dataset, model supports training at up to 0.2B parameters and achieves 54.71% MFU with near-linear scalability (0.97).