Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection
Not on the map. Why?
Abstract (arXiv)
The critical damping condition of the damped harmonic oscillator model of SGD with momentum (Qian, 1999) yields a momentum schedule with no tuned hyperparameters: mu(t) = 1 - 2*sqrt(alpha(t)). Across five seeds on ResNet-18/CIFAR-10 (200-epoch cosine schedule) it reaches 90% test accuracy 2.34x faster than constant mu=0.9 (range 1.71-2.86x, 5/5 seeds, one-sided paired t-test p=4e-4), at the cost of a real final-accuracy deficit of 0.46 pp (5/5 seeds, p=0.009). A short-schedule control rules out a schedule-length artifact: compressed baselines either pay 0.5-0.9 pp of accuracy or stay slower to 90% at equal accuracy. A hybrid recipe -- critical-damping momentum until 90%, then constant mu=0.9 -- removes the deficit and keeps the speedup: 95.45 +/- 0.05% final accuracy at 2.4x faster progress to 90% (n=5). The speedup generalizes across architectures (VGG-16 without skip connections: 1.72x, n=3); on CIFAR-100 early gains persist (2-4x to mid-training thresholds) but the accuracy cost grows (-1.7 pp), compressing the accuracy-matched gain to 1.14x. We also report an exhaustive negative result on surgical layer selection. Version 2 of this paper claimed that gradient attribution on misclassified images selects which layers to retrain; running the identical correction protocol on all 35 combinations of 3-of-7 layer groups ranks the selected triple 11th of 35 (exact p=0.31) -- no better than random. What survives is weaker but real: combinations containing the top-ranked layer outperform the rest (+6.2 vs -1.4 mean net error reduction), and the bottom of the gradient-norm ranking reliably predicts the most harmful interventions (down to -20 net errors). Gradient attribution on errors is a harm-avoidance signal, not a selector of repair targets. We release the full 35-combination landscape as a baseline for layer-selection claims.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 30 Mar 2026.
Builds on
16 of 26 references.
References · 16
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2022Mass-Editing Memory in a Transformercited by 976
- 2015Cyclical Learning Rates for Training Neural Networkscited by 627
- 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
- 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
- 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
- 2018Understanding the Acceleration Phenomenon via High-Resolution Differential Equationscited by 68
- 2017YellowFin and the Art of Momentum Tuningcited by 16
- 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
- 2019Demon: Improved Neural Network Training with Momentum Decaycited by 11
4 more
- 2021Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimizationcited by 8
- 2025From SGD to Spectra: A Theory of Neural Network Weight Dynamicscited by 8
- 2026Adaptive Momentum and Nonlinear Damping for Neural Network Trainingcited by 3
- 2025Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Modelscited by 1
Built on by
Not cited yet.