mlsanityData to 7 Oct 2026

Modular Block-diagonal Curvature Approximations for Feedforward Architectures

arXiv 1902.01813 ↗First posted 5 Feb 2019cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian). The approach reduces the otherwise tedious manual derivation of these matrices into local modules, and is easy to integrate into existing machine learning libraries. Moreover, we develop a compact notation derived from matrix differential calculus. We outline different strategies applicable to our method. They subsume recently-proposed block-diagonal approximations as special cases, and are extended to convolutional neural networks in this work.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 1; 2024: 3; 2025: 4; 2026: 3.

13 papers.

Sort:

  1. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  2. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  3. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  4. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  5. 2024What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysiscited by 9
  6. 2023ISAAC Newton: Input-based Approximate Curvature for Newton's Methodcited by 6
  7. 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
  8. 2026Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvaturecited by 3
  9. 2025Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functionscited by 2
  10. 2025Local properties of neural networks through the lens of layer-wise Hessianscited by 2
  11. 2025CurvaDion: Curvature-Adaptive Distributed Orthonormalizationcited by 1
  12. 2026Efficient Bilevel Optimization with KFAC-Based Hypergradientscited by 1
1 more
  1. 2026Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compressioncited by 0