# Wukong: Towards a Scaling Law for Large-Scale Recommendation

arXiv: https://arxiv.org/abs/2403.02545 · HTML: https://mlsanity.com/paper/2403.02545/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://mlsanity.com/paper/1803.05170/) (2018)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [BARS: Towards Open Benchmarking for Recommender Systems](https://mlsanity.com/paper/2205.09626/) (2022)
- [Open Benchmarking for Click-Through Rate Prediction](https://mlsanity.com/paper/2009.05794/) (2020)
- [Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions](https://mlsanity.com/paper/1909.03276/) (2019)
- [MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask](https://mlsanity.com/paper/2102.07619/) (2021)
- [VIP5: Towards Multimodal Foundation Models for Recommendation](https://mlsanity.com/paper/2305.14302/) (2023)
- [Higher-Order Factorization Machines](https://mlsanity.com/paper/1607.07195/) (2016)
- [Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python](https://mlsanity.com/paper/2112.08429/) (2021)
- [Learning to Embed Categorical Features without Embedding Tables for Recommendation](https://mlsanity.com/paper/2010.10784/) (2020)
- [Understanding Scaling Laws for Recommendation Models](https://mlsanity.com/paper/2208.08489/) (2022)
- [FinalMLP: An Enhanced Two-Stream MLP Model for CTR Prediction](https://mlsanity.com/paper/2304.00902/) (2023)
- [Monolith: Real Time Recommendation System With Collisionless Embedding Table](https://mlsanity.com/paper/2209.07663/) (2022)
- [On the Embedding Collapse when Scaling up Recommendation Models](https://mlsanity.com/paper/2310.04400/) (2023)
- [Scaling Law for Recommendation Models: Towards General-purpose User Representations](https://mlsanity.com/paper/2111.11294/) (2021)
- [Scaling Law of Large Sequential Recommendation Models](https://mlsanity.com/paper/2311.11351/) (2023)
- [Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training](https://mlsanity.com/paper/1805.07891/) (2018)
- [Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters](https://mlsanity.com/paper/2111.05897/) (2021)
- [Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems](https://mlsanity.com/paper/2311.05884/) (2023)
- [Improving Training Stability for Multitask Ranking Models in Recommender Systems](https://mlsanity.com/paper/2302.09178/) (2023)

## Built on by (most cited first, 30 of 74)

- [Scaling Optimal LR Across Token Horizons](https://mlsanity.com/paper/2409.19913/) (2024)
- [SLMRec: Distilling Large Language Models into Small for Sequential Recommendation](https://mlsanity.com/paper/2405.17890/) (2024)
- [MiniOneRec: An Open-Source Framework for Scaling Generative Recommendation](https://mlsanity.com/paper/2510.24431/) (2025)
- [External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation](https://mlsanity.com/paper/2502.17494/) (2025)
- [GR2: Generative Reasoning Re-ranker](https://mlsanity.com/paper/2602.07774/) (2026)
- [The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exit](https://mlsanity.com/paper/2501.02173/) (2025)
- [InterFormer: Effective Heterogeneous Interaction Learning for Click-Through Rate Prediction](https://mlsanity.com/paper/2411.09852/) (2024)
- [Kunlun: Establishing Scaling Laws for Massive-Scale Recommendation Systems through Unified Architecture Design](https://mlsanity.com/paper/2602.10016/) (2026)
- [AgentX: Towards Agent-Driven Self-Iteration of Industrial Recommender Systems](https://mlsanity.com/paper/2606.26859/) (2026)
- [UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems](https://mlsanity.com/paper/2604.00590/) (2026)
- [Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics](https://mlsanity.com/paper/2505.00347/) (2025)
- [Yambda-5B -- A Large-Scale Multi-modal Dataset for Ranking And Retrieval](https://mlsanity.com/paper/2505.22238/) (2025)
- [KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta](https://mlsanity.com/paper/2512.23236/) (2025)
- [LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation](https://mlsanity.com/paper/2601.20083/) (2026)
- [Exploring Training and Inference Scaling Laws in Generative Retrieval](https://mlsanity.com/paper/2503.18941/) (2025)
- [Large Language Model Empowered Recommendation Meets All-domain Continual Pre-Training](https://mlsanity.com/paper/2504.08949/) (2025)
- [Field Matters: A Lightweight LLM-enhanced Method for CTR Prediction](https://mlsanity.com/paper/2505.14057/) (2025)
- [Realizing Scaling Laws in Recommender Systems: A Foundation-Expert Paradigm for Hyperscale Model Deployment](https://mlsanity.com/paper/2508.02929/) (2025)
- [PACEvolve++: Improving Test-time Learning for Evolutionary Search Agents](https://mlsanity.com/paper/2605.07039/) (2026)
- [Scaling Laws for Online Advertisement Retrieval](https://mlsanity.com/paper/2411.13322/) (2024)
- [On the Practice of Deep Hierarchical Ensemble Network for Ad Conversion Rate Prediction](https://mlsanity.com/paper/2504.08169/) (2025)
- [MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation](https://mlsanity.com/paper/2510.15286/) (2025)
- [Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts](https://mlsanity.com/paper/2510.25285/) (2025)
- [From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction](https://mlsanity.com/paper/2511.12081/) (2025)
- [TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds](https://mlsanity.com/paper/2604.13737/) (2026)
- [ReasonRec: A Reasoning-Augmented Multimodal Agent for Unified Recommendation](https://mlsanity.com/paper/2606.28357/) (2026)
- [BehaveGPT: A Foundation Model for Large-scale User Behavior Modeling](https://mlsanity.com/paper/2505.17631/) (2025)
- [RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models](https://mlsanity.com/paper/2509.20883/) (2025)
- [Adaptive Regularization for Large-Scale Sparse Feature Embedding Models](https://mlsanity.com/paper/2511.06374/) (2025)
- [Hi-SAM: A Hierarchical Structure-Aware Multi-modal Framework for Large-Scale Recommendation](https://mlsanity.com/paper/2602.11799/) (2026)

## Abstract (arXiv)

Scaling laws play an instrumental role in the sustainable improvement in model quality. Unfortunately, recommendation models to date do not exhibit such laws similar to those observed in the domain of large language models, due to the inefficiencies of their upscaling mechanisms. This limitation poses significant challenges in adapting these models to increasingly more complex real-world datasets. In this paper, we propose an effective network architecture based purely on stacked factorization machines, and a synergistic upscaling strategy, collectively dubbed Wukong, to establish a scaling law in the domain of recommendation. Wukong's unique design makes it possible to capture diverse, any-order of interactions simply through taller and wider layers. We conducted extensive evaluations on six public datasets, and our results demonstrate that Wukong consistently outperforms state-of-the-art models quality-wise. Further, we assessed Wukong's scalability on an internal, large-scale dataset. The results show that Wukong retains its superiority in quality over state-of-the-art models, while holding the scaling law across two orders of magnitude in model complexity, extending beyond 100 GFLOP/example, where prior arts fall short.