mlsanityData to 7 Oct 2026

Inverse Depth Scaling From Most Layers Being Similar

arXiv 2602.05970 ↗First posted 5 Feb 2026cs.LG cs.AI math.DScited by 8

Not on the map. Why?

Abstract (arXiv)

Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies. Here, we quantify how depth affects loss via analysis of LLMs and toy residual networks. We find loss scales inversely proportional to depth in LLMs, probably due to functionally similar layers reducing error through ensemble averaging rather than compositional learning or discretizing smooth dynamics. This regime is inefficient yet robust and may arise from the architectural bias of residual networks and target functions incompatible with smooth dynamics. The findings suggest that improving LLM efficiency may require architectural innovations to encourage compositional use of depth.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

38 of 46 references.

References · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  10. 2022Mass-Editing Memory in a Transformercited by 976
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  12. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
26 more
  1. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  2. 2021Explaining Neural Scaling Lawscited by 353
  3. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  4. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  5. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  6. 2023The Quantization Model of Neural Scalingcited by 137
  7. 2022A Solvable Model of Neural Scaling Lawscited by 101
  8. 2021Learning Curve Theorycited by 79
  9. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  10. 2024Chinchilla Scaling: A replication attemptcited by 65
  11. 2016Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networkscited by 58
  12. 2025The Curse of Depth in Large Language Modelscited by 58
  13. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  14. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  15. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  16. 2025Superposition Yields Robust Neural Scalingcited by 30
  17. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  18. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  19. 2025How Do LLMs Use Their Depth?cited by 15
  20. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 13
  21. 2024When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Modelscited by 12
  22. 2024Looking Beyond The Top-1: Transformers Determine Top Tokens In Ordercited by 12
  23. 2024A Resource Model For Neural Scaling Lawcited by 8
  24. 2025Physics of Skill Learningcited by 8
  25. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 8
  26. 2025What Affects the Effective Depth of Large Language Models?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoningcited by 12
  2. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficientscited by 4
  3. 2026Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depthcited by 2
  4. 2026The Model Shape Behind Scaling Laws: A Gradient Superposition Perspectivecited by 0
  5. 2026Emergent One-Third Scaling Law as Attention Tries to Concentratecited by 0
  6. 2026DepthBench: Measuring How Residual Connections Enable More Computational Depthcited by 0
  7. 2026Emergent phases of superposition: from partial to full representationcited by 0
  8. 2026Optimizer-dependent training dynamics converge to the same one-third optimal data scalingcited by 0