mlsanityData to 7 Oct 2026

AdaFisher: Adaptive Second Order Optimization via Fisher Information

arXiv 2405.16397 ↗First posted 26 May 2024cs.LG math.OCcited by 14

Not on the map. Why?

Abstract (arXiv)

First-order optimization methods are currently the mainstream in training deep neural networks (DNNs). Optimizers like Adam incorporate limited curvature information by employing the diagonal matrix preconditioning of the stochastic gradient during the training. Despite their widespread, second-order optimization algorithms exhibit superior convergence properties compared to their first-order counterparts e.g. Adam and SGD. However, their practicality in training DNNs is still limited due to increased per-iteration computations compared to the first-order methods. We present AdaFisher--an adaptive second-order optimizer that leverages a diagonal block-Kronecker approximation of the Fisher information matrix for adaptive gradient preconditioning. AdaFisher aims to bridge the gap between enhanced convergence/generalization capabilities and computational efficiency in second-order optimization framework for training DNNs. Despite the slow pace of second-order optimizers, we showcase that AdaFisher can be reliably adopted for image classification, language modeling and stands out for its stability and robustness in hyper-parameter tuning. We demonstrate that AdaFisher outperforms the SOTA optimizers in terms of both accuracy and convergence speed. Code is available from https://github.com/AtlasAnalyticsLab/AdaFisher.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 53 of its 91 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

53 of 91 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2016Layer Normalizationcited by 5,579
  8. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2021Convex optimizationcited by 2,479
  11. 2019Searching for MobileNetV3cited by 1,984
  12. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
38 more
  1. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2016An overview of gradient descent optimization algorithmscited by 1,021
  4. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  6. 2019On the Convergence of Adam and Beyondcited by 834
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  8. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  10. 2017Large Batch Training of Convolutional Networkscited by 529
  11. 2014New insights and perspectives on the natural gradient methodcited by 435
  12. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  13. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  14. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  15. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  16. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  17. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  18. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  19. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  20. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  21. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  22. 2022Focal Modulation Networkscited by 186
  23. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  24. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  25. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  26. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  27. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  28. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  29. 2018Data Augmentation using Random Image Cropping and Patching for Deep CNNscited by 87
  30. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  31. 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30
  32. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  33. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  34. 2022How Does Adaptive Optimization Impact Local Neural Network Geometry?cited by 24
  35. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 20
  36. 2022Depth-Wise Attention (DWAtt): A Layer Fusion Method for Data-Efficient Classificationcited by 9
  37. 2019Inefficiency of K-FAC for Large Batch Size Trainingcited by 4
  38. 2024Scalable Continuous-time Diffusion Framework for Network Inference and Influence Estimationcited by 4

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 8.

14 papers.

Sort:

  1. 2025Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulatorcited by 13
  2. 2026SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Trainingcited by 5
  3. 2025Stochastic Approximation with Block Coordinate Optimal Stepsizescited by 3
  4. 2025SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximationcited by 2
  5. 2025KOALA++: Efficient Kalman-Based Optimization with Gradient-Covariance Productscited by 2
  6. 2024A Mechanistic Study of Transformers Training Dynamicscited by 1
  7. 2026Dynamic Momentum Recalibration in Online Gradient Learningcited by 1
  8. 2026Layerwise LQR for Geometry-Aware Optimization of Deep Networkscited by 1
  9. 2026Information-Geometric Forward Policy Training in GFlowNetscited by 1
  10. 2025Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Trickscited by 0
  11. 2026Motivation is Something You Needcited by 0
  12. 2026Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limitcited by 0
2 more
  1. 2026Fisher-Guided Progressive Parameter Selection for Adaptive Fine-Tuningcited by 0
  2. 2026On the second-order optimization for spiking neural networkscited by 0