Realizing Scaling Laws in Recommender Systems: A Foundation-Expert Paradigm for Hyperscale Model Deployment
On the map: Knowledge distillation → Distillation for recommender systems
Abstract (arXiv)
Scaling laws have been established for recommender systems, yet efficiently deploying foundation model (FM) across multiple recommendation surfaces remains a major unsolved challenge. Existing methods for transfer learning face fundamental limitations in this setting: knowledge distillation suffers from transfer fidelity degradation in the large-data regime, and static user or item embeddings lack the expressiveness to capture contextualized user-item interactions. We propose the Foundation-Expert paradigm, where a central FM generates target-aware embeddings which are ingested by lightweight surface-specific expert models. Target-aware embeddings are representations that dynamically capture a user's interest in a specific item conditioned on their full interaction history. Unlike knowledge distillation, which transfers FM knowledge as soft labels, our approach provides these embeddings as input features to each expert model, enabling direct interaction with surface-specific representations. This paradigm achieves transfer ratios of 0.64--1.0 from FM to experts, substantially exceeding existing methods. Fully deployed at Meta serving tens of billions of daily requests since 2025, it delivers 0.050% statistically significant online topline metric improvement and 0.359% cumulative gains across multiple surfaces.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Distillation for recommender systems, one of 12 directions in Knowledge distillation. Builds on 26 of its 41 references that are in the map. First posted on arXiv 4 Aug 2025.
Builds on
26 of 41 references.
References · 26
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2020Knowledge Distillation: A Surveycited by 1,603
- 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
- 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
- 2017Learning multiple visual domains with residual adapterscited by 557
14 more
- 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
- 2024Foundation Models for Time Series Analysis: A Tutorial and Surveycited by 239
- 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 188
- 2019Practice on Long Sequential User Behavior Modeling for Click-Through Rate Predictioncited by 118
- 2023Modular Deep Learningcited by 83
- 2024Wukong: Towards a Scaling Law for Large-Scale Recommendationcited by 74
- 2025Distillation Scaling Lawscited by 67
- 2025MTGR: Industrial-Scale Generative Recommendation Framework in Meituancited by 49
- 2025LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenderscited by 47
- 2022ItemSage: Learning Product Embeddings for Shopping Recommendations at Pinterestcited by 23
- 2023Scaling User Modeling: Large-scale Online User Representations for Ads Personalization in Metacited by 21
- 2025External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendationcited by 13
- 2020General-Purpose User Embeddings based on Mobile App Usagecited by 11
- 2024Efficient user history modeling with amortized inference for deep learning recommendation modelscited by 5
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026A Unified Language Model for Large Scale Search, Recommendation, and Reasoningcited by 2
- 2026Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scalecited by 0
- 2026OneShot: Index-in-Ranking with Neural Scoring for Large-Scale Retrievalcited by 0
- 2026Progressive Alignment of Recommender Foundation Model through Multi-Phase Post-Trainingcited by 0
- 2026Progressive Alignment of Recommender Foundation Model through Multi-Phase Post-Training7 Aug 2026
- 2026OneShot: Index-in-Ranking with Neural Scoring for Large-Scale Retrieval29 Jul 2026
- 2026Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale27 Apr 2026
- 2026A Unified Language Model for Large Scale Search, Recommendation, and Reasoning18 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms
- 2023Embedding in Recommender Systems: A Survey
- 2025Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts
- 2025Improving LLM Interpretability and Performance via Guided Embedding Refinement for Sequential Recommendation
- 2025Bridging Collaborative Filtering and Large Language Models with Dynamic Alignment, Multimodal Fusion and Evidence-grounded Explanations
- 2023Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations
- 2026Inherit4Rec: Parameter Inheritance for Efficient Scaling of Recommendation Models ↗
- 2026SMES: Towards Scalable Multi-Task Recommendation via Expert Sparsity