mlsanityData to 14 Oct 2026

Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning

arXiv 2501.07400 ↗First posted 13 Jan 2025cs.LG cs.AI math.APcited by 0

Not on the map. Why?

Abstract (arXiv)

We derive explicit equations governing the cumulative biases and weights in Deep Learning with ReLU activation function, based on gradient descent for the Euclidean loss in the input layer, and under the assumption that the weights are, in a precise sense, adapted to the coordinate system distinguished by the activations. We show that gradient descent corresponds to a dynamical process in the input layer, whereby clusters of data are progressively reduced in complexity ("truncated") at an exponential rate that increases with the number of data points that have already been truncated. We provide a detailed discussion of several types of solutions to the gradient flow equations. A main motivation for this work is to shed light on the interpretability question in supervised learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 13 Jan 2025.

Builds on

15 of 21 references.

References · 15

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
  3. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  4. 2016Deep Learning without Poor Local Minimacited by 234
  5. 2014Qualitatively characterizing neural network optimization problemscited by 188
  6. 2020Neural collapse with unconstrained featurescited by 142
  7. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  8. 2020On the emergence of simplex symmetry in the final and penultimate layers of neural network classifierscited by 52
  9. 2022Functional dimension of feedforward ReLU neural networkscited by 21
  10. 2023Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscapecited by 17
  11. 2023Geometric structure of Deep Learning networks and construction of global mathcal L2 minimizerscited by 6
  12. 2023Geometric structure of shallow neural networks and constructive mathcal L2 cost minimizationcited by 4
3 more
  1. 2024Interpretable global minima of deep ReLU neural networks on sequentially separable datacited by 4
  2. 2023Global L2 minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learningcited by 2
  3. 2024Gradient flow in parameter space is equivalent to linear interpolation in output spacecited by 2

Built on by

Not cited yet.