mlsanityData to 7 Oct 2026

Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

arXiv 2402.17152 ↗First posted 27 Feb 2024cs.LG cs.IRcited by 184

Not on the map. Why?

Abstract (arXiv)

Large-scale recommendation systems are characterized by their reliance on high cardinality, heterogeneous features and the need to handle tens of billions of user actions on a daily basis. Despite being trained on huge volume of data with thousands of features, most Deep Learning Recommendation Models (DLRMs) in industry fail to scale with compute. Inspired by success achieved by Transformers in language and vision domains, we revisit fundamental design choices in recommendation systems. We reformulate recommendation problems as sequential transduction tasks within a generative modeling framework ("Generative Recommenders"), and propose a new architecture, HSTU, designed for high cardinality, non-stationary streaming recommendation data. HSTU outperforms baselines over synthetic and public datasets by up to 65.8% in NDCG, and is 5.3x to 15.2x faster than FlashAttention2-based Transformers on 8192 length sequences. HSTU-based Generative Recommenders, with 1.5 trillion parameters, improve metrics in online A/B tests by 12.4% and have been deployed on multiple surfaces of a large internet platform with billions of users. More importantly, the model quality of Generative Recommenders empirically scales as a power-law of training compute across three orders of magnitude, up to GPT-3/LLaMa-2 scale, which reduces carbon footprint needed for future model developments, and further paves the way for the first foundational models in recommendations.

Key facts. Cited by 184 papers in the map (data through 7 Oct 2026). Builds on 49 of its 68 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

49 of 68 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
37 more
  1. 2018Self-Attentive Sequential Recommendationcited by 1,169
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2016Deep Networks with Stochastic Depthcited by 973
  4. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
  5. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  6. 2020On Layer Normalization in the Transformer Architecturecited by 906
  7. 2016Wide & Deep Learning for Recommender Systemscited by 745
  8. 2015Session-based Recommendations with Recurrent Neural Networkscited by 685
  9. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 673
  10. 2017Deep Interest Network for Click-Through Rate Predictioncited by 541
  11. 2023TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendationcited by 336
  12. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 326
  13. 2020S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximizationcited by 311
  14. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  15. 2022Transformer Quality in Linear Timecited by 189
  16. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  17. 2022M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systemscited by 152
  18. 2023Text Is All You Need: Learning Language Representations for Sequential Recommendationcited by 149
  19. 2019Behavior Sequence Transformer for E-commerce Recommendation in Alibabacited by 128
  20. 2017Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networkscited by 126
  21. 2020Neural Collaborative Filtering vs. Matrix Factorization Revisitedcited by 123
  22. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  23. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 99
  24. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  25. 2021FBGEMM: Enabling High-Performance Low-Precision Deep Learning Inferencecited by 40
  26. 2023TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishoucited by 40
  27. 2023TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterestcited by 35
  28. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32
  29. 2020COLD: Towards the Next Generation of Pre-Ranking Systemcited by 30
  30. 2023Turning Dross Into Gold Loss: is BERT4Rec really better than SASRec?cited by 30
  31. 2021Zero-Shot Recommendation as Language Modelingcited by 28
  32. 2020Learning Optimal Tree Models Under Beam Searchcited by 26
  33. 2021Scaling Law for Recommendation Models: Towards General-purpose User Representationscited by 26
  34. 2021A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Modelscited by 24
  35. 2014Expectation-Maximization for Learning Determinantal Point Processescited by 19
  36. 2023Revisiting Neural Retrieval on Acceleratorscited by 14
  37. 2023Breaking the Curse of Quality Saturation with User-Centric Rankingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 56; 2026: 112.

184 papers.

Sort:

By area:

  1. 2024Wukong: Towards a Scaling Law for Large-Scale Recommendationcited by 72
  2. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  3. 2024HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modelingcited by 63
  4. 2025Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representationscited by 35
  5. 2025Relational Graph Transformercited by 30
  6. 2025PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendationscited by 27
  7. 2024Towards Next-Generation LLM-based Recommender Systems: A Survey and Beyondcited by 26
  8. 2025iAgent: LLM Agent as a Shield between User and Recommender Systemscited by 23
  9. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  10. 2024SLMRec: Distilling Large Language Models into Small for Sequential Recommendationcited by 18
  11. 2024Language Representations Can be What Recommenders Need: Findings and Potentialscited by 18
  12. 2025ActionPiece: Contextually Tokenizing Action Sequences for Generative Recommendationcited by 18
38 more
  1. 2025RecoWorld: Building Simulated Environments for Agentic Recommender Systemscited by 18
  2. 2025MiniOneRec: An Open-Source Framework for Scaling Generative Recommendationcited by 16
  3. 2025MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptationcited by 10
  4. 2025CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendationcited by 10
  5. 2025Massive Memorization with Hundreds of Trillions of Parameters for Sequential Transducer Generative Recommenderscited by 9
  6. 2026Why Thinking Hurts: Diagnosing and Rectifying Linguistic Inertia in Large Language Models for Recommendationcited by 9
  7. 2024Preference Diffusion for Recommendationcited by 8
  8. 2025PinRec: Unified Generative Retrieval for Pinterest Recommender Systemscited by 8
  9. 2025PinFM: Foundation Model for User Activity Sequences at a Billion-scale Visual Discovery Platformcited by 8
  10. 2026Kunlun: Establishing Scaling Laws for Massive-Scale Recommendation Systems through Unified Architecture Designcited by 8
  11. 2026OneReason Technical Reportcited by 8
  12. 2026AgentX: Towards Agent-Driven Self-Iteration of Industrial Recommender Systemscited by 8
  13. 2026Generative Recommendation for Large-Scale Advertisingcited by 7
  14. 2026UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systemscited by 7
  15. 2024Generate and Instantiate What You Prefer: Text-Guided Diffusion for Sequential Recommendationcited by 6
  16. 2025Understanding Generative Recommendation with Semantic IDs from a Model-scaling Viewcited by 6
  17. 2025Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamicscited by 5
  18. 2025TBGRecall: A Generative Retrieval Model for E-commerce Recommendation Scenarioscited by 5
  19. 2026LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendationcited by 5
  20. 2026DeepInterestGR: Mining Deep Multi-Interest Using Multi-Modal LLMs for Generative Recommendationcited by 5
  21. 2025Large Language Model as Universal Retriever in Industrial-Scale Recommender Systemcited by 4
  22. 2025Multi-Behavior Recommender Systems: A Surveycited by 4
  23. 2025Pre-training Generative Recommender with Multi-Identifier Item Tokenizationcited by 4
  24. 2025OmniSage: Large Scale, Multi-Entity Heterogeneous Graph Representation Learningcited by 4
  25. 2025Act-With-Think: Chunk Auto-Regressive Modeling for Generative Recommendationcited by 4
  26. 2025Realizing Scaling Laws in Recommender Systems: A Foundation-Expert Paradigm for Hyperscale Model Deploymentcited by 4
  27. 2025PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modelingcited by 4
  28. 2025DiffGRM: Diffusion-based Generative Recommendation Modelcited by 4
  29. 2026Variable-Length Semantic IDs for Recommender Systemscited by 4
  30. 2026HyTRec: A Hybrid Temporal-Aware Attention Architecture for Long Behavior Sequential Recommendationcited by 4
  31. 2026Agentic Recommender System with Hierarchical Belief-State Memorycited by 4
  32. 2026SafeGEO: Understanding Generative Engine Optimization Risks in Recommendation Agentscited by 4
  33. 2024Retrieval with Learned Similaritiescited by 3
  34. 2024Scaling Laws for Online Advertisement Retrievalcited by 3
  35. 2025Graph Foundation Models for Recommendation: A Comprehensive Surveycited by 3
  36. 2025GRACE: Generative Recommendation via Journey-Aware Sparse Attention on Chain-of-Thought Tokenizationcited by 3
  37. 2025eSASRec: Enhancing Transformer-based Recommendations in a Modular Fashioncited by 3
  38. 2025xLLM Technical Reportcited by 3

50 most cited of 184 All 184 (JSON)