# On-line learning dynamics of ReLU neural networks using statistical physics techniques

arXiv: https://arxiv.org/abs/1903.07378 · HTML: https://mlsanity.com/paper/1903.07378/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2019.

## Built on by (most cited first, 4 of 4)

- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [High-dimensional learning of narrow neural networks](https://mlsanity.com/paper/2409.13904/) (2024)
- [Analytic theory of dropout regularization](https://mlsanity.com/paper/2505.07792/) (2025)
- [An Analytical Theory of Auxiliary Learning](https://mlsanity.com/paper/2609.29774/) (2026)

## Abstract (arXiv)

We introduce exact macroscopic on-line learning dynamics of two-layer neural networks with ReLU units in the form of a system of differential equations, using techniques borrowed from statistical physics. For the first experiments, numerical solutions reveal similar behavior compared to sigmoidal activation researched in earlier work. In these experiments the theoretical results show good correspondence with simulations. In ove-rrealizable and unrealizable learning scenarios, the learning behavior of ReLU networks shows distinctive characteristics compared to sigmoidal networks.