mlsanityData to 14 Oct 2026

FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information

arXiv 2405.12807 ↗First posted 21 May 2024cs.LG cs.AI cs.ITcited by 18

Not on the map. Why?

Abstract (arXiv)

This paper establishes a mathematical foundation for the Adam optimizer, elucidating its connection to natural gradient descent through Riemannian and information geometry. We provide an accessible and detailed analysis of the diagonal empirical Fisher information matrix (FIM) in Adam, clarifying all detailed approximations and advocating for the use of log probability functions as loss, which should be based on discrete distributions, due to the limitations of empirical FIM. Our analysis uncovers flaws in the original Adam algorithm, leading to proposed corrections such as enhanced momentum calculations, adjusted bias corrections, adaptive epsilon, and gradient clipping. We refine the weight decay term based on our theoretical framework. Our modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 45 of its 77 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

45 of 77 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  12. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
33 more
  1. 2023Mastering Diverse Domains through World Modelscited by 1,253
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  3. 2019On the Convergence of Adam and Beyondcited by 841
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  5. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  8. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  9. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  10. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  11. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  12. 2021GFlowNet Foundationscited by 290
  13. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  14. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  15. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  16. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  17. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  18. 2013Revisiting Natural Gradient for Deep Networkscited by 159
  19. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  20. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 141
  21. 2020MetNet: A Neural Weather Model for Precipitation Forecastingcited by 131
  22. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  23. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  24. 2018Improving Regression Performance with Distributional Lossescited by 67
  25. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 40
  26. 2018Fisher Information and Natural Gradient Learning of Random Deep Networkscited by 28
  27. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  28. 2020LaProp: Separating Momentum and Adaptivity in Adamcited by 20
  29. 2022The Inverse of Exact Renormalization Group Flows as Statistical Inferencecited by 12
  30. 2020EAdam Optimizer: How ε Impact Adamcited by 10
  31. 2022On the Dynamics of Inference and Learningcited by 9
  32. 2024Batch size invariant Adamcited by 5
  33. 2024Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 7.

18 papers.

Sort:

By area:

  1. 2024SWAN: SGD with Normalization and Whitening Enables Stateless LLM Trainingcited by 20
  2. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 19
  3. 2025Beyond the LUMIR challenge: The pathway to foundational registration modelscited by 12
  4. 2025Forget Forgetting: Continual Learning in a World of Abundant Memorycited by 11
  5. 2025Stochastic Approximation with Block Coordinate Optimal Stepsizescited by 3
  6. 2025Module-Aware Parameter-Efficient Machine Unlearning on Transformerscited by 3
  7. 2025Toward a Holistic Approach to Continual Model Mergingcited by 3
  8. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3
  9. 2024High-precision medical speech recognition through synthetic data and semantic correction: UNITED-MEDASRcited by 2
  10. 2025How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networkscited by 1
  11. 2025Preconditioned Regularized Wasserstein Proximal Samplingcited by 1
  12. 2026Natural Riemannian gradient for learning functional tensor networkscited by 1
6 more
  1. 2024Fine-Tuning TransMorph with Gradient Correlation for Anatomical Alignmentcited by 0
  2. 2026Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratiocited by 0
  3. 2026MAdam: Metric-Aware Multi-Objective Adamcited by 0
  4. 2026Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimizationcited by 0
  5. 2026Fisher8: Stabilizing Neural Heteroscedastic Regression via Output-Layer Fisher Geometrycited by 0
  6. 2026How Far is Adam from Natural Gradient Descent?cited by 0