mlsanityData to 14 Oct 2026

The Model Shape Behind Scaling Laws: A Gradient Superposition Perspective

arXiv 2605.27989 ↗First posted 27 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural scaling laws guide resource allocation, but how should a fixed parameter budget be divided between depth and width? Motivated by inverse-width representation-error scaling under strong superposition, we study this allocation through the network's end-to-end response. The Neural Feature Ansatz links weight and gradient geometry, motivating the Frobenius norm F of the input average gradient outer product (AGOP) as an observable of response strength and overlap. By reproducing Anthropic's toy-model double-descent experiment, we find that F and test reconstruction loss follow an approximately affine relationship across the full sample-size sweep and share an interpolation peak. Through Gaussian minimum-norm regression experiments, we further show that this correlation arises from a fitted-noise amplification term shared by prediction error and F. By pretraining language models spanning different shapes and parameter budgets, we find a strong positive correlation between loss and F among the best shapes selected by validation loss. This supports interpreting effective parameter allocation as preserving task-relevant computation while reducing the amplification of error responses. Our results identify error-response amplification as a mechanism linking gradient superposition to generalization, providing a functional account of depth-width allocation and efficient parameter utilization. The code is availale at: hyperlinkhttps://github.com/wenjie1835/ShapeScalinghttps://github.com/wenjie1835/ShapeScaling.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

31 of 37 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2022Toy Models of Superpositioncited by 971
  8. 2018Universal Transformerscited by 647
  9. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  10. 2024Scaling and evaluating sparse autoencoderscited by 553
  11. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  12. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
19 more
  1. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  2. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  3. 2023Scaling Data-Constrained Language Modelscited by 403
  4. 2021Explaining Neural Scaling Lawscited by 363
  5. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  6. 2016Benefits of depth in neural networkscited by 250
  7. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  8. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
  9. 2016Robust Large Margin Deep Neural Networkscited by 118
  10. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  11. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 38
  12. 2025Superposition Yields Robust Neural Scalingcited by 31
  13. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 25
  14. 2026How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Modelscited by 24
  15. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  16. 2024Feature learning as alignment: a structural property of gradient descent in non-linear neural networkscited by 10
  17. 2026Inverse Depth Scaling From Most Layers Being Similarcited by 9
  18. 2025On the Neural Feature Ansatz for Deep Neural Networkscited by 2
  19. 2026Revisiting the Shape Convention of Transformer Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines
  2. 2026The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Laws
  3. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
  4. 2026Emergent phases of superposition: from partial to full representation
  5. 2026Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Model
  6. 2025Superposition as Lossy Compression: Measure with Sparse Autoencoders and Connect to Adversarial Vulnerability
  7. 2026Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective
  8. 2026Feature Superposition in Neural Networks: From Theory to Practice