mlsanityData to 14 Oct 2026

ASDL: A Unified Interface for Gradient Preconditioning in PyTorch

arXiv 2305.04684 ↗First posted 8 May 2023cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Gradient preconditioning is a key technique to integrate the second-order information into gradients for improving and extending gradient-based learning algorithms. In deep learning, stochasticity, nonconvexity, and high dimensionality lead to a wide variety of gradient preconditioning methods, with implementation complexity and inconsistent performance and feasibility. We propose the Automatic Second-order Differentiation Library (ASDL), an extension library for PyTorch, which offers various implementations and a plug-and-play unified interface for gradient preconditioning. ASDL enables the study and structured comparison of a range of gradient preconditioning methods.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 34 of its 69 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

34 of 69 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  8. 2017Large Batch Training of Convolutional Networkscited by 535
  9. 2014New insights and perspectives on the natural gradient methodcited by 443
  10. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  11. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  12. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
22 more
  1. 2018An Empirical Model of Large-Batch Trainingcited by 281
  2. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  3. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  4. 2013Revisiting Natural Gradient for Deep Networkscited by 159
  5. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  6. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 142
  7. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  8. 2019Quantum Natural Gradientcited by 106
  9. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 105
  10. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  11. 2019NGBoost: Natural Gradient Boosting for Probabilistic Predictioncited by 94
  12. 2019BackPACK: Packing more into backpropcited by 78
  13. 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 68
  14. 2015Preconditioned Stochastic Gradient Descentcited by 50
  15. 2018Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networkscited by 37
  16. 2019Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networkscited by 33
  17. 2013Riemannian metrics for neural networks I: feedforward networkscited by 30
  18. 2020Optimization of Graph Neural Networks with Natural Gradient Descentcited by 26
  19. 2021Tensor Normal Training for Deep Learning Modelscited by 24
  20. 2020Scalable and Practical Natural Gradient for Large-Scale Deep Learningcited by 15
  21. 2020Sketchy Empirical Natural Gradient Methods for Deep Learningcited by 11
  22. 2017Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networkscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 8; 2025: 4; 2026: 5.

21 papers.

Sort:

By area:

  1. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 115
  2. 2024A New Perspective on Shampoo's Preconditionercited by 59
  3. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  4. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  5. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 23
  6. 2025The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newtoncited by 17
  7. 2023The Memory Perturbation Equation: Understanding Model's Sensitivity to Datacited by 16
  8. 2024Curvature-Informed SGD via General Purpose Lie-Group Preconditionerscited by 16
  9. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 15
  10. 2024AdaFisher: Adaptive Second Order Optimization via Fisher Informationcited by 14
  11. 2024Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihoodcited by 7
  12. 2025Beyond the Mean: Fisher-Orthogonal Projection for Natural Gradient Descent in Large Batch Trainingcited by 5
9 more
  1. 2024Stochastic Hessian Fittings with Lie Groupscited by 4
  2. 2024PETScML: Second-order solvers for training regression problems in Scientific Machine Learningcited by 4
  3. 2025Understanding SOAP from the Perspective of Gradient Whiteningcited by 2
  4. 2026Efficient Bilevel Optimization with KFAC-Based Hypergradientscited by 1
  5. 2026Runtime-Orchestrated Second-Order Optimization for Scalable LLM Trainingcited by 1
  6. 2026Detecting Contaminated Code-Generation Prompt Batches via Influence Functionscited by 1
  7. 2024Stein Variational Newton Neural Network Ensemblescited by 0
  8. 2026Bayesian Fine-tuning in Projected Subspacescited by 0
  9. 2026Steering by Influence: Curvature Aware Data Weighting for Activation Steeringcited by 0