mlsanityData to 14 Oct 2026

Learning curves theory for hierarchically compositional data with power-law distributed features

arXiv 2505.07067 ↗First posted 11 May 2025stat.ML cond-mat.dis-nn cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Recent theories suggest that Neural Scaling Laws arise whenever the task is linearly decomposed into power-law distributed units. Alternatively, scaling laws also emerge when data exhibit a hierarchically compositional structure, as is thought to occur in language and images. To unify these views, we consider classification and next-token prediction tasks based on probabilistic context-free grammars -- probabilistic models that generate data via a hierarchy of production rules. For classification, we show that having power-law distributed production rules results in a power-law learning curve with an exponent depending on the rules' distribution and a large multiplicative constant that depends on the hierarchical structure. By contrast, for next-token prediction, the distribution of production rules controls the local details of the learning curve, but not the exponent describing the large-scale behaviour.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 11 May 2025.

Builds on

31 of 43 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  5. 2021Explaining Neural Scaling Lawscited by 363
  6. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  7. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  8. 2023The Quantization Model of Neural Scalingcited by 138
  9. 2022A Solvable Model of Neural Scaling Lawscited by 116
  10. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  11. 2024A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Datacited by 93
  12. 2021Learning Curve Theorycited by 85
19 more
  1. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  2. 2024Transformers represent belief state geometry in their residual streamcited by 77
  3. 2020Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networkscited by 72
  4. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 71
  5. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
  6. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48
  7. 2024Towards a theory of how the structure of language is acquired by deep neural networkscited by 39
  8. 2024Transformers Can Represent n-gram Language Modelscited by 32
  9. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  10. 2024Understanding Transformers via N-gram Statisticscited by 26
  11. 2024Probing the Latent Hierarchical Structure of Data via Diffusion Modelscited by 26
  12. 2018A Provably Correct Algorithm for Deep Learning that Actually Workscited by 20
  13. 2024What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languagescited by 20
  14. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  15. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Modelscited by 17
  16. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  17. 2025A Statistical Theory of Contrastive Pre-training and Multimodal Generative AIcited by 13
  18. 2024Can Transformers Learn n-gram Language Models?cited by 11
  19. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 15.

17 papers.

Sort:

By area:

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  2. 2025Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Modelscited by 25
  3. 2026Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentioncited by 10
  4. 2026Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chainingcited by 6
  5. 2026Scale Dependent Data Duplicationcited by 4
  6. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
  7. 2026Incremental Learning of Sparse Attention Patterns in Transformerscited by 2
  8. 2026Sampling Data with Chains of Forward-Backward Diffusion Stepscited by 2
  9. 2026Statistical Properties of Training & Generalizationcited by 2
  10. 2026Diffusion Models Preferentially Memorize Prototypical Examples or: Why Does My Diffusion Model Love Slop?cited by 1
  11. 2026Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tailcited by 1
  12. 2026Smooth Scaling Laws Hide Stepwise Token Learningcited by 1
5 more
  1. 2025Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-trainingcited by 0
  2. 2026Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networkscited by 0
  3. 2026Critical Percolation as a Synthetic Data Model for Interpretabilitycited by 0
  4. 2026Limits of Reliability and Scaling in Language Modelscited by 0
  5. 2026Neural Dynamics as the Composition of Quantized Unitscited by 0