mlsanityData to 7 Oct 2026

A Resource Model For Neural Scaling Law

arXiv 2402.05164 ↗First posted 7 Feb 2024cs.LG cs.AI cs.NEcited by 8

Not on the map. Why?

Abstract (arXiv)

Neural scaling laws characterize how model performance improves as the model size scales up. Inspired by empirical observations, we introduce a resource model of neural scaling. A task is usually composite hence can be decomposed into many subtasks, which compete for resources (measured by the number of neurons allocated to subtasks). On toy problems, we empirically find that: (1) The loss of a subtask is inversely proportional to its allocated neurons. (2) When multiple subtasks are present in a composite task, the resources acquired by each subtask uniformly grow as models get larger, keeping the ratios of acquired resources constants. We hypothesize these findings to be generally true and build a model to predict neural scaling laws for general composite tasks, which successfully replicates the neural scaling law of Chinchilla models reported in arXiv:2203.15556. We believe that the notion of resource used in this paper will be a useful tool for characterizing and diagnosing neural networks.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

22 of 30 references.

References · 22

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  5. 2022In-context Learning and Induction Headscited by 930
  6. 2022Toy Models of Superpositioncited by 928
  7. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  10. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  11. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  12. 2021Explaining Neural Scaling Lawscited by 353
10 more
  1. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  2. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  3. 2023The Quantization Model of Neural Scalingcited by 137
  4. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  5. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  6. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  7. 2020A Neural Scaling Law from the Dimension of the Data Manifoldcited by 48
  8. 2021Redundant representations help generalization in wide neural networkscited by 9
  9. 2023A Neural Scaling Law from Lottery Ticket Ensemblingcited by 5
  10. 2023The semantic landscape paradigm for neural networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 2.

8 papers.

  1. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  2. 2025Superposition Yields Robust Neural Scalingcited by 30
  3. 2026On the origin of neural scaling laws: from random graphs to natural languagecited by 12
  4. 2025Physics of Skill Learningcited by 8
  5. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 8
  6. 2024Survival of the Fittest Representation: A Case Study with Modular Additioncited by 7
  7. 2024AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Lawscited by 7
  8. 2024Scaling Laws for Post Training Quantized Large Language Modelscited by 4