mlsanityData to 7 Oct 2026

Revisiting Natural Gradient for Deep Networks

arXiv 1301.3584 ↗First posted 16 Jan 2013cs.LG cs.NA math.NAcited by 157

Not on the map. Why?

Abstract (arXiv)

We evaluate natural gradient, an algorithm originally proposed in Amari (1997), for learning deep models. The contributions of this paper are as follows. We show the connection between natural gradient and three other recently proposed methods for training deep models: Hessian-Free (Martens, 2010), Krylov Subspace Descent (Vinyals and Povey, 2012) and TONGA (Le Roux et al., 2008). We describe how one can use unlabeled data to improve the generalization error obtained by natural gradient and empirically evaluate the robustness of the algorithm to the ordering of the training set compared to stochastic gradient descent. Finally we extend natural gradient to incorporate second order information alongside the manifold information and provide a benchmark of the new algorithm using a truncated Newton approach for inverting the metric matrix instead of using a diagonal approximation of it.

Key facts. Cited by 157 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jan 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 27; 2023: 21; 2024: 33; 2025: 32; 2026: 24.

157 papers.

Sort:

By area:

  1. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  2. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  3. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  4. 2022Language model compression with weighted low-rank factorizationcited by 192
  5. 2021The Bayesian Learning Rulecited by 95
  6. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  7. 2024Fisher Flow Matching for Generative Modeling over Discrete Datacited by 68
  8. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  9. 2024A New Perspective on Shampoo's Preconditionercited by 54
  10. 2024Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizercited by 51
  11. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  12. 2021Posterior Meta-Replay for Continual Learningcited by 45
38 more
  1. 2021Bootstrapped Meta-Learningcited by 45
  2. 2021A Bayesian Federated Learning Framework with Online Laplace Approximationcited by 41
  3. 2023Weighted Ensemble Models Are Strong Continual Learnerscited by 41
  4. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 30
  5. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  6. 2023Elastic Weight Removal for Faithful and Abstractive Dialogue Generationcited by 29
  7. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  8. 2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
  9. 2022Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularizationcited by 23
  10. 2021On the Variance of the Fisher Information for Deep Learningcited by 22
  11. 2024Data Attribution for Text-to-Image Models by Unlearning Synthesized Imagescited by 22
  12. 2022Mitigating Dataset Bias by Using Per-sample Gradientcited by 21
  13. 2025Synthetic Data is an Elegant GIFT for Continual Vision-Language Modelscited by 21
  14. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 20
  15. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuningcited by 19
  16. 2022Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptivelycited by 18
  17. 2022Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithmcited by 17
  18. 2024FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher informationcited by 17
  19. 2021Task-agnostic Continual Learning with Hybrid Probabilistic Modelscited by 16
  20. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  21. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  22. 2022Hierarchical Memory Learning for Fine-Grained Scene Graph Generationcited by 15
  23. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  24. 2022Numerical Optimizations for Weighted Low-rank Estimation on Language Modelcited by 14
  25. 2023Automatic Gradient Descent: Deep Learning without Hyperparameterscited by 14
  26. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14
  27. 2022GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalizationcited by 13
  28. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  29. 2024LiRank: Industrial Large Scale Ranking Models at LinkedIncited by 12
  30. 2025Wasserstein Policy Optimizationcited by 12
  31. 2022DeepStability: A Study of Unstable Numerical Methods and Their Solutions in Deep Learningcited by 11
  32. 2022DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learningcited by 11
  33. 2021FedLGA: Towards System-Heterogeneity of Federated Learning via Local Gradient Approximationcited by 10
  34. 2023Achieving High Accuracy with PINNs via Energy Natural Gradientscited by 10
  35. 2024Trade-Offs of Diagonal Fisher Information Matrix Estimatorscited by 10
  36. 2022Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decompositioncited by 9
  37. 2024Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgettingcited by 9
  38. 2022Few-Shot Learning by Dimensionality Reduction in Gradient Spacecited by 8

50 most cited of 157 All 157 (JSON)