mlsanityData to 14 Oct 2026

Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos

arXiv 2605.21648 ↗First posted 20 May 2026cs.LG cond-mat.dis-nn cs.NEcited by 0

Not on the map. Why?

Abstract (arXiv)

We ask whether the standard treatment of dropout as a static hyperparameter is optimal, or whether its utility can be improved by letting it vary over depth. We answer this by developing a mean-field theory of dropout near the edge of chaos, identifying distinct universality classes for smooth and kinked activations, together with their scaling exponents. The resulting propagation theory, constrained by maximizing the regularization delivered by dropout, motivates concentrating dropout near the input. Experiments on vision, speech and financial time series show gains most consistently in MLPs, with smaller gains in Transformers.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 35 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

19 of 35 references.

References · 19

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2016Deep Networks with Stochastic Depthcited by 979
  4. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  5. 2017Deep Neural Networks as Gaussian Processescited by 622
  6. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  7. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  8. 2016Deep Information Propagationcited by 195
  9. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  10. 2021The Principles of Deep Learning Theorycited by 173
  11. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  12. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
7 more
  1. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  2. 2013Dropout Training as Adaptive Regularizationcited by 93
  3. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  4. 2017Curriculum Dropoutcited by 50
  5. 2017Benchmark Dataset for Mid-Price Forecasting of Limit Order Book Data with Machine Learning Methodscited by 49
  6. 2019Mean field theory for deep dropout networks: digging up gradient backpropagation deeplycited by 4
  7. 2023Les Houches Lectures on Deep Learning at Large & Infinite Widthcited by 1

Built on by

Not cited yet.