# Geometrical structures of digital fluctuations in parameter space of neural networks trained with adaptive momentum optimization

arXiv: https://arxiv.org/abs/2408.12273 · HTML: https://mlsanity.com/paper/2408.12273/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Builds on (most cited first, 10 of 10)

- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [A Theory on Adam Instability in Large-Scale Machine Learning](https://mlsanity.com/paper/2304.09871/) (2023)
- [Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problem](https://mlsanity.com/paper/2101.08286/) (2021)
- [On the loss landscape of a class of deep neural networks with no bad local valleys](https://mlsanity.com/paper/1809.10749/) (2018)
- [Adaptive Learning Rate Clipping Stabilizes Learning](https://mlsanity.com/paper/1906.09060/) (2019)
- [Stability for the Training of Deep Neural Networks and Other Classifiers](https://mlsanity.com/paper/2002.04122/) (2020)
- [Limitations of neural network training due to numerical instability of backpropagation](https://mlsanity.com/paper/2210.00805/) (2022)
- [Stabilizing Backpropagation in 16-bit Neural Training with Modified Adam Optimizer](https://mlsanity.com/paper/2307.16189/) (2023)
- [Algorithms and data structures for numerical computations with automatic precision estimation](https://mlsanity.com/paper/2403.16660/) (2024)

## Built on by (most cited first, 1 of 1)

- [Algorithms and data structures for automatic precision estimation of neural networks](https://mlsanity.com/paper/2509.24607/) (2025)

## Abstract (arXiv)

We present results of numerical experiments for neural networks with stochastic gradient-based optimization with adaptive momentum. This widely applied optimization has proved convergence and practical efficiency, but for long-run training becomes numerically unstable. We show that numerical artifacts are observable not only for large-scale models and finally lead to divergence also for case of shallow narrow networks. We argue this theory by experiments with more than 1600 neural networks trained for 50000 epochs. Local observations show presence of the same behavior of network parameters in both stable and unstable training segments. Geometrical behavior of parameters forms double twisted spirals in the parameter space and is caused by alternating of numerical perturbations with next relaxation oscillations in values for 1st and 2nd momentum.