# Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection

arXiv: https://arxiv.org/abs/2603.28921 · HTML: https://mlsanity.com/paper/2603.28921/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [Surgical Fine-Tuning Improves Adaptation to Distribution Shifts](https://mlsanity.com/paper/2210.11466/) (2022)
- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [Understanding the Acceleration Phenomenon via High-Resolution Differential Equations](https://mlsanity.com/paper/1810.08907/) (2018)
- [YellowFin and the Art of Momentum Tuning](https://mlsanity.com/paper/1706.03471/) (2017)
- [The Marginal Value of Momentum for Small Learning Rate SGD](https://mlsanity.com/paper/2307.15196/) (2023)
- [Demon: Improved Neural Network Training with Momentum Decay](https://mlsanity.com/paper/1910.04952/) (2019)
- [Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization](https://mlsanity.com/paper/2110.09057/) (2021)
- [From SGD to Spectra: A Theory of Neural Network Weight Dynamics](https://mlsanity.com/paper/2507.12709/) (2025)
- [Adaptive Momentum and Nonlinear Damping for Neural Network Training](https://mlsanity.com/paper/2602.00334/) (2026)
- [Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models](https://mlsanity.com/paper/2510.12044/) (2025)

## Abstract (arXiv)

The critical damping condition of the damped harmonic oscillator model of SGD with momentum (Qian, 1999) yields a momentum schedule with no tuned hyperparameters: mu(t) = 1 - 2*sqrt(alpha(t)). Across five seeds on ResNet-18/CIFAR-10 (200-epoch cosine schedule) it reaches 90% test accuracy 2.34x faster than constant mu=0.9 (range 1.71-2.86x, 5/5 seeds, one-sided paired t-test p=4e-4), at the cost of a real final-accuracy deficit of 0.46 pp (5/5 seeds, p=0.009). A short-schedule control rules out a schedule-length artifact: compressed baselines either pay 0.5-0.9 pp of accuracy or stay slower to 90% at equal accuracy. A hybrid recipe -- critical-damping momentum until 90%, then constant mu=0.9 -- removes the deficit and keeps the speedup: 95.45 +/- 0.05% final accuracy at 2.4x faster progress to 90% (n=5). The speedup generalizes across architectures (VGG-16 without skip connections: 1.72x, n=3); on CIFAR-100 early gains persist (2-4x to mid-training thresholds) but the accuracy cost grows (-1.7 pp), compressing the accuracy-matched gain to 1.14x. We also report an exhaustive negative result on surgical layer selection. Version 2 of this paper claimed that gradient attribution on misclassified images selects which layers to retrain; running the identical correction protocol on all 35 combinations of 3-of-7 layer groups ranks the selected triple 11th of 35 (exact p=0.31) -- no better than random. What survives is weaker but real: combinations containing the top-ranked layer outperform the rest (+6.2 vs -1.4 mean net error reduction), and the bottom of the gradient-norm ranking reliably predicts the most harmful interventions (down to -20 net errors). Gradient attribution on errors is a harm-avoidance signal, not a selector of repair targets. We release the full 35-combination landscape as a baseline for layer-selection claims.