# On the weight dynamics of learning networks

arXiv: https://arxiv.org/abs/2405.00743 · HTML: https://mlsanity.com/paper/2405.00743/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 0 of its 3 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Abstract (arXiv)

Neural networks have become a widely adopted tool for tackling a variety of problems in machine learning and artificial intelligence. In this contribution we use the mathematical framework of local stability analysis to gain a deeper understanding of the learning dynamics of feed forward neural networks. Therefore, we derive equations for the tangent operator of the learning dynamics of three-layer networks learning regression tasks. The results are valid for an arbitrary numbers of nodes and arbitrary choices of activation functions. Applying the results to a network learning a regression task, we investigate numerically, how stability indicators relate to the final training-loss. Although the specific results vary with different choices of initial conditions and activation functions, we demonstrate that it is possible to predict the final training loss, by monitoring finite-time Lyapunov exponents or covariant Lyapunov vectors during the training process.