# Demon: Improved Neural Network Training with Momentum Decay

arXiv: https://arxiv.org/abs/1910.04952 · HTML: https://mlsanity.com/paper/1910.04952/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2019.

## Built on by (most cited first, 11 of 11)

- [FAIR-FATE: Fair Federated Learning with Momentum](https://mlsanity.com/paper/2209.13678/) (2022)
- [Exact Gauss-Newton Optimization for Training Deep Neural Networks](https://mlsanity.com/paper/2405.14402/) (2024)
- [Enhancing Deep Neural Network Training Efficiency and Performance through Linear Prediction](https://mlsanity.com/paper/2310.10958/) (2023)
- [MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization](https://mlsanity.com/paper/2606.17526/) (2026)
- [Better Schedules for Low Precision Training of Deep Neural Networks](https://mlsanity.com/paper/2403.02243/) (2024)
- [Incremental Gauss-Newton Descent for Machine Learning](https://mlsanity.com/paper/2408.05560/) (2024)
- [Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis](https://mlsanity.com/paper/2508.03105/) (2025)
- [Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimization](https://mlsanity.com/paper/2510.04988/) (2025)
- [Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolution](https://mlsanity.com/paper/2404.06679/) (2024)
- [Critical Damping as a Momentum Schedule: Multi-Seed Validation, a Hybrid Recipe, and an Exhaustive Negative Result on Surgical Layer Selection](https://mlsanity.com/paper/2603.28921/) (2026)
- [A Physical Response-and-Memory Model for Muon Optimization](https://mlsanity.com/paper/2608.22994/) (2026)

## Abstract (arXiv)

Momentum is a widely used technique for gradient-based optimizers in deep learning. In this paper, we propose a decaying momentum (textscDemon) rule. We conduct the first large-scale empirical analysis of momentum decay methods for modern neural network optimization, in addition to the most popular learning rate decay schedules. Across 28 relevant combinations of models, epochs, datasets, and optimizers, textscDemon achieves the highest number of Top-1 and Top-3 finishes at 39% and 85% respectively, almost doubling the second-placed learning rate cosine schedule at 17% and 60%, respectively. textscDemon also outperforms other widely used schedulers including, but not limited to, the learning rate step schedule, linear schedule, OneCycle schedule, and exponential schedule. Compared with the widely used learning rate step schedule, textscDemon is observed to be less sensitive to parameter tuning, which is critical to training neural networks in practice. Results are demonstrated across a variety of settings and architectures, including image classification, generative models, and language models. textscDemon is easy to implement, requires no additional tuning, and incurs almost no extra computational overhead compared to the vanilla counterparts. Code is readily available.