mlsanityData to 7 Oct 2026

Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time

arXiv 2510.01098 ↗First posted 1 Oct 2025stat.ML cond-mat.dis-nn cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

We study in-context learning (ICL) of linear regression in a deep linear self-attention model, characterizing how performance depends on various computational and statistical resources (width, depth, number of training steps, batch size and data per context). In a joint limit where data dimension, context length, and residual stream width scale proportionally, we analyze the limiting asymptotics for three ICL settings: (1) isotropic covariates and tasks (ISO), (2) fixed and structured covariance (FS), and (3) where covariances are randomly rotated and structured (RRS). For ISO and FS settings, we find that depth only aids ICL performance if context length is limited. Alternatively, in the RRS setting where covariances change across contexts, increasing the depth leads to significant improvements in ICL, even at infinite context length. This provides a new solvable toy model of neural scaling laws which depends on both width and depth of a transformer and predicts an optimal transformer shape as a function of compute. This toy model enables computation of exact asymptotics for the risk as well as derivation of powerlaws under source/capacity conditions for the ICL tasks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 61 of its 69 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

61 of 69 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  8. 2022Transformers learn in-context by gradient descentcited by 714
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  12. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
38 more
  1. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  2. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  3. 2021Explaining Neural Scaling Lawscited by 353
  4. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  5. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  6. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  7. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  8. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  9. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  10. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
  11. 2025Hierarchical Reasoning Modelcited by 167
  12. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  13. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  14. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  15. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  16. 2022A Solvable Model of Neural Scaling Lawscited by 101
  17. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  18. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  19. 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 71
  20. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  21. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  22. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  23. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  24. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  25. 2024Asymptotic theory of in-context learning by linear attentioncited by 54
  26. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  27. 2024ICLR: In-Context Learning of Representationscited by 49
  28. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 49
  29. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  30. 2024Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscited by 40
  31. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  32. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 36
  33. 2024LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling lawcited by 32
  34. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  35. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
  36. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  37. 2025Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networkscited by 21
  38. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 19

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 8
  2. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 8
  3. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficientscited by 4
  4. 2026Superposition unifies power-law training dynamicscited by 1
  5. 2026Polynomial Speedup in Diffusion Models with the Multilevel Euler-Maruyama Methodcited by 1
  6. 2026Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fitscited by 0
  7. 2026Adaptivity Under Realizability Constraints: Comparing In-Context and Agentic Learningcited by 0
  8. 2026An Asymptotic Theory of Chain-of-Thought in In-Context Learningcited by 0
  9. 2026When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformerscited by 0
  10. 2026Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatchcited by 0
  11. 2026Deep neural networks as lattice gauge theoriescited by 0
  12. 2026Landau theory of quenched criticality in linear in-context learningcited by 0
3 more
  1. 2026The Capability Manifold and ML Scaling Lawscited by 0
  2. 2026Emergent One-Third Scaling Law as Attention Tries to Concentratecited by 0
  3. 2026What can linear attention learn from nonlinear teachers in-context?cited by 0