mlsanityData to 7 Oct 2026

HesScale: Scalable Computation of Hessian Diagonals

arXiv 2210.11639 ↗First posted 20 Oct 2022cs.LG math.OCcited by 8

Not on the map. Why?

Abstract (arXiv)

Second-order optimization uses curvature information about the objective function, which can help in faster convergence. However, such methods typically require expensive computation of the Hessian matrix, preventing their usage in a scalable way. The absence of efficient ways of computation drove the most widely used methods to focus on first-order approximations that do not capture the curvature information. In this paper, we develop HesScale, a scalable approach to approximating the diagonal of the Hessian matrix, to incorporate second-order information in a computationally efficient manner. We show that HesScale has the same computational complexity as backpropagation. Our results on supervised classification show that HesScale achieves high approximation accuracy, allowing for scalable and efficient second-order optimization.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 22 of its 39 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

22 of 39 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Layer Normalizationcited by 5,579
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  6. 2019On the Convergence of Adam and Beyondcited by 834
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  8. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  9. 2019Importance Estimation for Neural Network Pruningcited by 531
  10. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  11. 2014New insights and perspectives on the natural gradient methodcited by 435
  12. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
10 more
  1. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  2. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  3. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  4. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  5. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  6. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  7. 2019BackPACK: Packing more into backpropcited by 77
  8. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
  9. 2019DeepOBS: A Deep Learning Optimizer Benchmark Suitecited by 37
  10. 2019On the Convergence Proof of AMSGrad and a New Versioncited by 21

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 3.

8 papers.

  1. 2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
  2. 2024Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancementcited by 43
  3. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  4. 2024Trade-Offs of Diagonal Fisher Information Matrix Estimatorscited by 10
  5. 2025A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearningcited by 6
  6. 2023Utility-based Perturbed Gradient Descent: An Optimizer for Continual Learningcited by 5
  7. 2025Estimating the Probability of Sampling a Trained Neural Network at Randomcited by 3
  8. 2025Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methodscited by 1