mlsanityData to 14 Oct 2026

On the distance between two neural networks and the stability of learning

arXiv 2002.03432 ↗First posted 9 Feb 2020cs.LG cs.NA cs.NEcited by 46

Not on the map. Why?

Abstract (arXiv)

This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems to require little to no learning rate tuning, it may unlock a simpler workflow for training deeper and more complex neural networks. The Python code used in this paper is here: https://github.com/jxbz/fromage.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 7; 2023: 12; 2024: 5; 2025: 10; 2026: 5.

46 papers.

Sort:

By area:

  1. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  2. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  3. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  4. 2023A Spectral Condition for Feature Learningcited by 109
  5. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  6. 2024Scalable Optimization in the Modular Normcited by 68
  7. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  8. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  9. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 39
  10. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23
  11. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  12. 2025Why Gradients Rapidly Increase Near the End of Trainingcited by 22
34 more
  1. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  2. 2023Automatic Gradient Descent: Deep Learning without Hyperparameterscited by 14
  3. 2021Deep Bayesian inference for seismic imaging with taskscited by 12
  4. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  5. 2023Efficient Parametric Approximations of Neural Network Function Space Distancecited by 7
  6. 2021KOALA: A Kalman Optimization Algorithm with Loss Adaptivitycited by 6
  7. 2026Stabilizing Native Low-Rank LLM Pretrainingcited by 6
  8. 2023On the Weight Dynamics of Deep Normalized Networkscited by 4
  9. 2024A Differential Geometric View and Explainability of GNN on Evolving Graphscited by 4
  10. 2025Celo: Training Versatile Learned Optimizers on a Compute Dietcited by 4
  11. 2025Function-Space Learning Ratescited by 4
  12. 2025Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methodscited by 4
  13. 2022Learning to Optimize Quasi-Newton Methodscited by 3
  14. 2023On Suppressing Range of Adaptive Stepsizes of Adam to Improve Generalisation Performancecited by 3
  15. 2025Controlling changes to attention logitscited by 3
  16. 2022Optimisation & Generalisation in Networks of Neuronscited by 2
  17. 2023Federated Learning over a Wireless Network: Distributed User Selection through Random Accesscited by 2
  18. 2024Towards Simple and Provable Parameter-Free Adaptive Gradient Methodscited by 2
  19. 2025KOALA++: Efficient Kalman-Based Optimization with Gradient-Covariance Productscited by 2
  20. 2021Guiding Evolutionary Strategies by Differentiable Robot Simulatorscited by 1
  21. 2022A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Rangecited by 1
  22. 2022A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learningcited by 1
  23. 2023Multiplicative update rules for accelerating deep learning training and increasing robustnesscited by 1
  24. 2023Signal Processing Meets SGD: From Momentum to Filtercited by 1
  25. 2023Automatic Optimisation of Normalised Neural Networkscited by 1
  26. 2025HVAdam: A Full-Dimension Adaptive Optimizercited by 1
  27. 2026Dynamic Momentum Recalibration in Online Gradient Learningcited by 1
  28. 2021Solving hybrid machine learning tasks by traversing weight space geodesicscited by 0
  29. 2021Online-Learning Deep Neuro-Adaptive Dynamic Inversion Controller for Model Free Controlcited by 0
  30. 2022Towards understanding deep learning with the natural clustering priorcited by 0
  31. 2025Deep-Relative-Trust-Based Diffusion for Decentralized Deep Learningcited by 0
  32. 2025Deep Sturm--Liouville: From Sample-Based to 1D Regularization with Learnable Orthogonal Basis Functionscited by 0
  33. 2026Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samplescited by 0
  34. 2026RODE: A Radial-Orthogonal Decoupled Engine for Optimizationcited by 0