mlsanityData to 14 Oct 2026

The Depth Delusion: Why Transformers Should Be Wider, Not Deeper

arXiv 2601.20994 ↗First posted 28 Jan 2026cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Neural scaling laws describe how language model loss decreases with parameters and data, but treat architecture as interchangeable--a billion parameters could arise from a shallow-wide model (10 layers & 8,192 hidden dimension) or a deep-narrow one (80 layers & 2,048 hidden dimension). We propose architecture-conditioned scaling laws decomposing this dependence, finding that optimal depth scales as D* ~ C^0.12 while optimal width scales as W* ~ C^0.34, meaning width should grow 2.8x faster than depth. We discover a critical depth phenomenon: beyond D_crit ~ W^0.44 (sublinear in W), adding layers increases loss despite adding parameters--the Depth Delusion. Empirically, we validate these findings across 30 transformer architectures spanning 17M to 7B parameters, each trained on representative high-compute samples, achieving R^2 = 0.922. Our central finding: at 7B scale, a 64-layer model (6.38B params) underperforms a 32-layer model (6.86B params) by 0.12 nats, despite being significantly deeper. This demonstrates that optimal depth-width tradeoffs persist at the production scale.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 18 of its 22 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

18 of 22 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2023Mistral 7Bcited by 3,164
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  10. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  11. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  12. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
6 more
  1. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  2. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  3. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  4. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  5. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 106
  6. 2020The Depth-to-Width Interplay in Self-Attentioncited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depthcited by 2
  2. 2026Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scalingcited by 0