mlsanityData to 7 Oct 2026

Scaling Law for Recommendation Models: Towards General-purpose User Representations

arXiv 2111.11294 ↗First posted 15 Nov 2021cs.IR cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition and language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility of general-purpose user representation learning by training a universal user encoder at large scales. We demonstrate that the scaling law is present in user representation learning areas, where the training error scales as a power-law with the amount of computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, and the resulting user embeddings stretch our expectation of what is possible to do in various downstream tasks. CLUE also shows great transferability to other domains and companies, as performances on an online experiment shows significant improvements in Click-Through-Rate (CTR). Furthermore, we also investigate how the model performance is influenced by the scale factors, such as training data size, model capacity, sequence length, and batch size. Finally, we discuss the broader impacts of CLUE in general.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 15 Nov 2021.

Builds on

39 of 58 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
27 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2019Contrastive Multiview Codingcited by 1,297
  5. 2017Mixed Precision Trainingcited by 1,135
  6. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,134
  7. 2021Scaling Vision Transformerscited by 993
  8. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 673
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  10. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  11. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  12. 2021Explaining Neural Scaling Lawscited by 353
  13. 2020Representation Learning via Invariant Causal Mechanismscited by 197
  14. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  15. 2019Behavior Sequence Transformer for E-commerce Recommendation in Alibabacited by 128
  16. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  17. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  18. 2021Learning Curve Theorycited by 79
  19. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  20. 2018NSML: Meet the MLaaS platform with a real-world case studycited by 70
  21. 2017NSML: A Machine Learning Platform That Enables You to Focus on Your Modelscited by 34
  22. 2020One Person, One Model, One World: Learning Continual User Representation without Forgettingcited by 30
  23. 2020Exploiting Behavioral Consistence for Universal User Representationcited by 14
  24. 2021One4all User Representation for Recommender Systems in E-commercecited by 12
  25. 2021UserBERT: Contrastive User Model Pre-trainingcited by 6
  26. 2020div2vec: Diversity-Emphasized Node Embeddingcited by 3
  27. 2021Interest-oriented Universal User Representation via Contrastive Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 2; 2024: 9; 2025: 6; 2026: 5.

26 papers.

Sort:

By area:

  1. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  2. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  3. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 184
  4. 2024Wukong: Towards a Scaling Law for Large-Scale Recommendationcited by 72
  5. 2024HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modelingcited by 63
  6. 2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
  7. 2025(Mis)Fitting: A Survey of Scaling Lawscited by 24
  8. 2024Breaking the Length Barrier: LLM-Enhanced CTR Prediction in Long Textual User Behaviorscited by 23
  9. 2025External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendationcited by 13
  10. 2024Scaling Sequential Recommendation Models with Transformerscited by 7
  11. 2025Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Modelscited by 5
  12. 2022Nonlinear Initialization Methods for Low-Rank Neural Networkscited by 3
14 more
  1. 2022Learning Large-scale Universal User Representation with Sparse Mixture of Expertscited by 3
  2. 2024Scaling Laws for Online Advertisement Retrievalcited by 3
  3. 2022Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learningcited by 2
  4. 2025RecIS: Sparse to Dense, A Unified Training Framework for Recommendation Modelscited by 2
  5. 2023Self-Improving-Leaderboard(SIL): A Call for Real-World Centric Natural Language Processing Leaderboardscited by 1
  6. 2025On Model and Data Scaling for Skeleton-based Self-Supervised Gait Recognitioncited by 1
  7. 2025Catalog-Native LLM: Speaking Item-ID Dialect with Less Entanglement for Recommendationcited by 1
  8. 2026Hi-SAM: A Hierarchical Structure-Aware Multi-modal Framework for Large-Scale Recommendationcited by 1
  9. 2024Predicting Viral Rumors and Vulnerable Users for Infodemic Surveillancecited by 0
  10. 2024SoMeR: Multi-View User Representation Learning for Social Mediacited by 0
  11. 2026GRAB: An LLM-Inspired Sequence-First Click-Through Rate Prediction Modeling Paradigmcited by 0
  12. 2026Bumblebee: Interleaved Mixed-Layer Building Blocks for Large-Scale Recommendation Systemscited by 0
  13. 2026Don't Contrast the Impossible: Region-Constrained Batching for Contrastive User Modeling on a Local Community Platformcited by 0
  14. 2026Towards a Densing Law for User Representation Learning at Billion-Scale Capacitycited by 0