# Modular Block-diagonal Curvature Approximations for Feedforward Architectures

arXiv: https://arxiv.org/abs/1902.01813 · HTML: https://mlsanity.com/paper/1902.01813/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2019.

## Built on by (most cited first, 13 of 13)

- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [Tractable structured natural gradient descent using local parameterizations](https://mlsanity.com/paper/2102.07405/) (2021)
- [Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.15603/) (2024)
- [Towards Quantifying the Hessian Structure of Neural Networks](https://mlsanity.com/paper/2505.02809/) (2025)
- [What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis](https://mlsanity.com/paper/2410.10986/) (2024)
- [ISAAC Newton: Input-based Approximate Curvature for Newton's Method](https://mlsanity.com/paper/2305.00604/) (2023)
- [Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning](https://mlsanity.com/paper/2406.03276/) (2024)
- [Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions](https://mlsanity.com/paper/2509.23437/) (2025)
- [Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature](https://mlsanity.com/paper/2602.17385/) (2026)
- [Local properties of neural networks through the lens of layer-wise Hessians](https://mlsanity.com/paper/2510.17486/) (2025)
- [CurvaDion: Curvature-Adaptive Distributed Orthonormalization](https://mlsanity.com/paper/2512.13728/) (2025)
- [Efficient Bilevel Optimization with KFAC-Based Hypergradients](https://mlsanity.com/paper/2603.29108/) (2026)
- [Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression](https://mlsanity.com/paper/2609.02451/) (2026)

## Abstract (arXiv)

We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian). The approach reduces the otherwise tedious manual derivation of these matrices into local modules, and is easy to integrate into existing machine learning libraries. Moreover, we develop a compact notation derived from matrix differential calculus. We outline different strategies applicable to our method. They subsume recently-proposed block-diagonal approximations as special cases, and are extended to convolutional neural networks in this work.