# Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning

arXiv: https://arxiv.org/abs/2501.07400 · HTML: https://mlsanity.com/paper/2501.07400/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 13 Jan 2025.

## Builds on (most cited first, 15 of 15)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis](https://mlsanity.com/paper/1703.00560/) (2017)
- [On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers](https://mlsanity.com/paper/2012.05420/) (2020)
- [Functional dimension of feedforward ReLU neural networks](https://mlsanity.com/paper/2209.04036/) (2022)
- [Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape](https://mlsanity.com/paper/2305.19510/) (2023)
- [Geometric structure of Deep Learning networks and construction of global mathcal L2 minimizers](https://mlsanity.com/paper/2309.10639/) (2023)
- [Geometric structure of shallow neural networks and constructive mathcal L2 cost minimization](https://mlsanity.com/paper/2309.10370/) (2023)
- [Interpretable global minima of deep ReLU neural networks on sequentially separable data](https://mlsanity.com/paper/2405.07098/) (2024)
- [Global L2 minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learning](https://mlsanity.com/paper/2311.15487/) (2023)
- [Gradient flow in parameter space is equivalent to linear interpolation in output space](https://mlsanity.com/paper/2408.01517/) (2024)

## Abstract (arXiv)

We derive explicit equations governing the cumulative biases and weights in Deep Learning with ReLU activation function, based on gradient descent for the Euclidean loss in the input layer, and under the assumption that the weights are, in a precise sense, adapted to the coordinate system distinguished by the activations. We show that gradient descent corresponds to a dynamical process in the input layer, whereby clusters of data are progressively reduced in complexity ("truncated") at an exponential rate that increases with the number of data points that have already been truncated. We provide a detailed discussion of several types of solutions to the gradient flow equations. A main motivation for this work is to shed light on the interpretability question in supervised learning.