mlsanityData to 7 Oct 2026

Scaling Laws for Autoregressive Generative Modeling

arXiv 2010.14701 ↗First posted 28 Oct 2020cs.LG cs.CL cs.CVcited by 510

Not on the map. Why?

Cited by 33 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal imageleftrightarrowtext models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law. The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains. The cross-entropy loss has an information theoretic interpretation as S(True) + DKL(True||Model), and the empirical scaling laws suggest a prediction for both the true data distribution's entropy and the KL divergence between the true and model distributions. With this interpretation, billion-parameter Transformers are nearly perfect models of the YFCC100M image distribution downsampled to an 8× 8 resolution, and we can forecast the model size needed to achieve any given reducible loss (ie DKL) in nats/image for other resolutions. We find a number of additional scaling laws in specific domains: (a) we identify a scaling relation for the mutual information between captions and images in multimodal models, and show how to answer the question "Is a picture worth a thousand words?"; (b) in the case of mathematical problem solving, we identify scaling laws for model performance when extrapolating beyond the training distribution; (c) we finetune generative image models for ImageNet classification and find smooth scaling of the classification loss and error rate, even as the generative loss levels off. Taken together, these results strengthen the case that scaling laws have important implications for neural network performance, including on downstream tasks.

Key facts. Cited by 510 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 50; 2022: 45; 2023: 68; 2024: 130; 2025: 134; 2026: 83.

510 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  10. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  11. 2021Scaling Vision Transformerscited by 993
  12. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
38 more
  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  4. 2021Robust fine-tuning of zero-shot modelscited by 785
  5. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  8. 2024Yi: Open Foundation Models by 01.AIcited by 706
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  10. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  12. 2024Scaling and evaluating sparse autoencoderscited by 530
  13. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  14. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  15. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  16. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  17. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  18. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  19. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  20. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  21. 2023Scaling Data-Constrained Language Modelscited by 391
  22. 2021Explaining Neural Scaling Lawscited by 353
  23. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  24. 2023ClimaX: A foundation model for weather and climatecited by 325
  25. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  26. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  27. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  28. 2021Unsolved Problems in ML Safetycited by 291
  29. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  30. 2025Learning to Reason under Off-Policy Guidancecited by 266
  31. 2021Scaling Laws for Transfercited by 265
  32. 2021Generating Images with Sparse Representationscited by 262
  33. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  34. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  35. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  36. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  37. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  38. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199

50 most cited of 510 All 510 (JSON)