# Scaling Law for Recommendation Models: Towards General-purpose User Representations

arXiv: https://arxiv.org/abs/2111.11294 · HTML: https://mlsanity.com/paper/2111.11294/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 15 Nov 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Representation Learning via Invariant Causal Mechanisms](https://mlsanity.com/paper/2010.07922/) (2020)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1905.06874/) (2019)
- [Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup](https://mlsanity.com/paper/2101.06983/) (2021)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)

## Built on by (most cited first, 26 of 26)

- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling](https://mlsanity.com/paper/2409.12740/) (2024)
- [FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning](https://mlsanity.com/paper/2401.03230/) (2024)
- [(Mis)Fitting: A Survey of Scaling Laws](https://mlsanity.com/paper/2502.18969/) (2025)
- [Breaking the Length Barrier: LLM-Enhanced CTR Prediction in Long Textual User Behaviors](https://mlsanity.com/paper/2403.19347/) (2024)
- [External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation](https://mlsanity.com/paper/2502.17494/) (2025)
- [Scaling Sequential Recommendation Models with Transformers](https://mlsanity.com/paper/2412.07585/) (2024)
- [Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models](https://mlsanity.com/paper/2506.10972/) (2025)
- [Nonlinear Initialization Methods for Low-Rank Neural Networks](https://mlsanity.com/paper/2202.00834/) (2022)
- [Learning Large-scale Universal User Representation with Sparse Mixture of Experts](https://mlsanity.com/paper/2207.04648/) (2022)
- [Scaling Laws for Online Advertisement Retrieval](https://mlsanity.com/paper/2411.13322/) (2024)
- [Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learning](https://mlsanity.com/paper/2212.03760/) (2022)
- [On Model and Data Scaling for Skeleton-based Self-Supervised Gait Recognition](https://mlsanity.com/paper/2504.07598/) (2025)
- [RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Models](https://mlsanity.com/paper/2509.20883/) (2025)
- [Hi-SAM: A Hierarchical Structure-Aware Multi-modal Framework for Large-Scale Recommendation](https://mlsanity.com/paper/2602.11799/) (2026)
- [Self-Improving-Leaderboard(SIL): A Call for Real-World Centric Natural Language Processing Leaderboards](https://mlsanity.com/paper/2303.10888/) (2023)
- [Catalog-Native LLM: Speaking Item-ID Dialect with Less Entanglement for Recommendation](https://mlsanity.com/paper/2510.05125/) (2025)
- [Predicting Viral Rumors and Vulnerable Users for Infodemic Surveillance](https://mlsanity.com/paper/2401.09724/) (2024)
- [SoMeR: Multi-View User Representation Learning for Social Media](https://mlsanity.com/paper/2405.05275/) (2024)
- [GRAB: An LLM-Inspired Sequence-First Click-Through Rate Prediction Modeling Paradigm](https://mlsanity.com/paper/2602.01865/) (2026)
- [Bumblebee: Interleaved Mixed-Layer Building Blocks for Large-Scale Recommendation Systems](https://mlsanity.com/paper/2607.24804/) (2026)
- [Don't Contrast the Impossible: Region-Constrained Batching for Contrastive User Modeling on a Local Community Platform](https://mlsanity.com/paper/2607.28971/) (2026)
- [Towards a Densing Law for User Representation Learning at Billion-Scale Capacity](https://mlsanity.com/paper/2608.23392/) (2026)

## Abstract (arXiv)

Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition and language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility of general-purpose user representation learning by training a universal user encoder at large scales. We demonstrate that the scaling law is present in user representation learning areas, where the training error scales as a power-law with the amount of computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, and the resulting user embeddings stretch our expectation of what is possible to do in various downstream tasks. CLUE also shows great transferability to other domains and companies, as performances on an online experiment shows significant improvements in Click-Through-Rate (CTR). Furthermore, we also investigate how the model performance is influenced by the scale factors, such as training data size, model capacity, sequence length, and batch size. Finally, we discuss the broader impacts of CLUE in general.