mlsanityData to 7 Oct 2026

Directional convergence and alignment in deep learning

arXiv 2006.06657 ↗First posted 11 Jun 2020cs.LG cs.NE math.OCcited by 172

Not on the map. Why?

Abstract (arXiv)

In this paper, we show that although the minimizers of cross-entropy and related classification losses are off at infinity, network weights learned by gradient flow converge in direction, with an immediate corollary that network predictions, training errors, and the margin distribution also converge. This proof holds for deep homogeneous networks -- a broad class of networks allowing for ReLU, max-pooling, linear, and convolutional layers -- and we additionally provide empirical support not just close to the theory (e.g., the AlexNet), but also on non-homogeneous networks (e.g., the DenseNet). If the network further has locally Lipschitz gradients, we show that these gradients also converge in direction, and asymptotically align with the gradient flow path, with consequences on margin maximization, convergence of saliency maps, and a few other settings. Our analysis complements and is distinct from the well-known neural tangent and mean-field theories, and in particular makes no requirements on network width and initialization, instead merely requiring perfect classification accuracy. The proof proceeds by developing a theory of unbounded nonsmooth Kurdyka-Lojasiewicz inequalities for functions definable in an o-minimal structure, and is also applicable outside deep learning.

Key facts. Cited by 172 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 31; 2023: 35; 2024: 35; 2025: 31; 2026: 20.

172 papers.

Sort:

By area:

  1. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  2. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  3. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  4. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  5. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  6. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  7. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  8. 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
  9. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  10. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  11. 2023Transformers as Support Vector Machinescited by 67
  12. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
38 more
  1. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  2. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  3. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  4. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  5. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  6. 2021Nonsmooth Implicit Differentiation for Machine Learning and Optimizationcited by 43
  7. 2025Grokking at the Edge of Numerical Stabilitycited by 41
  8. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  9. 2022The Implicit Bias of Benign Overfittingcited by 38
  10. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  11. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  12. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  13. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  14. 2021Early-stopped neural networks are consistentcited by 34
  15. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  16. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  17. 2021Fast Margin Maximization via Dual Accelerationcited by 33
  18. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  19. 2023Feature emergence via margin maximization: case studies in algebraic taskscited by 33
  20. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  21. 2024The Implicit Bias of Adam on Separable Datacited by 32
  22. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  23. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  24. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  25. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  26. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  27. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  28. 2022Gradient Methods Provably Converge to Non-Robust Networkscited by 27
  29. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  30. 2023On the Implicit Bias of Adamcited by 25
  31. 2023Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Lossescited by 24
  32. 2023What's in a Prior? Learned Proximal Networks for Inverse Problemscited by 24
  33. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  34. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  35. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  36. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  37. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
  38. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21

50 most cited of 172 All 172 (JSON)