# A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks

arXiv: https://arxiv.org/abs/2010.13165 · HTML: https://mlsanity.com/paper/2010.13165/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2020.

## Built on by (most cited first, 12 of 12)

- [On the Convergence and Calibration of Deep Learning with Differential Privacy](https://mlsanity.com/paper/2106.07830/) (2021)
- [Sparse Neural Additive Model: Interpretable Deep Learning with Feature Selection via Group Sparsity](https://mlsanity.com/paper/2202.12482/) (2022)
- [MetaNODE: Prototype Optimization as a Neural ODE for Few-Shot Learning](https://mlsanity.com/paper/2103.14341/) (2021)
- [Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networks](https://mlsanity.com/paper/2107.01832/) (2021)
- [Covariate-Balancing-Aware Interpretable Deep Learning models for Treatment Effect Estimation](https://mlsanity.com/paper/2203.03185/) (2022)
- [A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networks](https://mlsanity.com/paper/2204.08306/) (2022)
- [HyperFlow: Gradient-Free Emulation of Few-Shot Fine-Tuning](https://mlsanity.com/paper/2504.15323/) (2025)
- [Scaling depth capacity via zero/one-layer model expansion](https://mlsanity.com/paper/2511.04981/) (2025)
- [Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks](https://mlsanity.com/paper/2208.03941/) (2022)
- [On the Convergence of Federated Averaging under Partial Participation for Over-parameterized Neural Networks](https://mlsanity.com/paper/2310.05495/) (2023)
- [Hybrid Coordinate Descent for Efficient Neural Network Learning Using Line Search and Gradient Descent](https://mlsanity.com/paper/2408.01374/) (2024)
- [Prototype Optimization with Neural ODE for Few-Shot Learning](https://mlsanity.com/paper/2411.12259/) (2024)

## Abstract (arXiv)

When equipped with efficient optimization algorithms, the over-parameterized neural networks have demonstrated high level of performance even though the loss function is non-convex and non-smooth. While many works have been focusing on understanding the loss dynamics by training neural networks with the gradient descent (GD), in this work, we consider a broad class of optimization algorithms that are commonly used in practice. For example, we show from a dynamical system perspective that the Heavy Ball (HB) method can converge to global minimum on mean squared error (MSE) at a linear rate (similar to GD); however, the Nesterov accelerated gradient descent (NAG) may only converges to global minimum sublinearly. Our results rely on the connection between neural tangent kernel (NTK) and finite over-parameterized neural networks with ReLU activation, which leads to analyzing the limiting ordinary differential equations (ODE) for optimization algorithms. We show that, optimizing the non-convex loss over the weights corresponds to optimizing some strongly convex loss over the prediction error. As a consequence, we can leverage the classical convex optimization theory to understand the convergence behavior of neural networks. We believe our approach can also be extended to other optimization algorithms and network architectures.