mlsanityData to 7 Oct 2026

Scaling Laws for Transfer

arXiv 2102.01293 ↗First posted 2 Feb 2021cs.LGcited by 265

Not on the map. Why?

Abstract (arXiv)

We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting. When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance (cross-entropy loss). When we do the same for models pre-trained on a large language dataset, the slope in performance gains is merely reduced rather than going to zero. We calculate the effective data "transferred" from pre-training by determining how much data a transformer of the same size would have required to achieve the same loss when training from scratch. In other words, we focus on units of data while holding everything else fixed. We find that the effective data transferred is described well in the low data regime by a power-law of parameter count and fine-tuning dataset size. We believe the exponents in these power-laws correspond to measures of the generality of a model and proximity of distributions (in a directed rather than symmetric sense). We find that pre-training effectively multiplies the fine-tuning dataset size. Transfer, like overall performance, scales predictably in terms of parameters, data, and compute.

Key facts. Cited by 265 papers in the map (data through 7 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 2 Feb 2021.

Builds on

22 of 45 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
10 more
  1. 2019Solving Rubik's Cube with a Robot Handcited by 699
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  3. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  4. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  5. 2018A Survey on Deep Transfer Learningcited by 417
  6. 2018Rethinking ImageNet Pre-trainingcited by 376
  7. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  8. 2015Semi-supervised Sequence Learningcited by 198
  9. 2020Measuring the Algorithmic Efficiency of Neural Networkscited by 51
  10. 2020A Neural Scaling Law from the Dimension of the Data Manifoldcited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 26; 2023: 46; 2024: 66; 2025: 60; 2026: 42.

265 papers.

Sort:

By area:

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2023BloombergGPT: A Large Language Model for Financecited by 923
  8. 2021Robust fine-tuning of zero-shot modelscited by 785
  9. 2023Large Language Modelscited by 774
  10. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  11. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  12. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
38 more
  1. 2023Scaling Data-Constrained Language Modelscited by 391
  2. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  3. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  4. 2025Small Language Models are the Future of Agentic AIcited by 254
  5. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  6. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  7. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  8. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  9. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  10. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  11. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  12. 2024On the Effects of Data Scale on UI Control Agentscited by 187
  13. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  14. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
  15. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  16. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  17. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  18. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  19. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  20. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  21. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  22. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  23. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  24. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  25. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  26. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  27. 2021Scaling Laws for Neural Machine Translationcited by 110
  28. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  29. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  30. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  31. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  32. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  33. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  34. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  35. 2022Broken Neural Scaling Lawscited by 102
  36. 2022A Solvable Model of Neural Scaling Lawscited by 101
  37. 2021The Efficiency Misnomercited by 98
  38. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91

50 most cited of 265 All 265 (JSON)