mlsanityData to 7 Oct 2026

Position: Curvature Matrices Should Be Democratized via Linear Operators

arXiv 2501.19183 ↗First posted 31 Jan 2025cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Structured large matrices are prevalent in machine learning. A particularly important class is curvature matrices like the Hessian, which are central to understanding the loss landscape of neural nets (NNs), and enable second-order optimization, uncertainty quantification, model pruning, data attribution, and more. However, curvature computations can be challenging due to the complexity of automatic differentiation, and the variety and structural assumptions of curvature proxies, like sparsity and Kronecker factorization. In this position paper, we argue that linear operators -- an interface for performing matrix-vector products -- provide a general, scalable, and user-friendly abstraction to handle curvature matrices. To support this position, we developed textitcurvlinops, a library that provides curvature matrices through a unified linear operator interface. We demonstrate with textitcurvlinops how this interface can hide complexity, simplify applications, be extensible and interoperable with other libraries, and scale to large NNs.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 49 of its 96 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

49 of 96 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  8. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  9. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  10. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  11. 2014New insights and perspectives on the natural gradient methodcited by 435
  12. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
37 more
  1. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  2. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  3. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  4. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  5. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  6. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  7. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  8. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  9. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  10. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  11. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  12. 2020Kernel Operations on the GPU, with Autodiff, without Memory Overflowscited by 101
  13. 2019BackPACK: Packing more into backpropcited by 77
  14. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 73
  15. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  16. 2020Hutch++: Optimal Stochastic Trace Estimationcited by 62
  17. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  18. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  19. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  20. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
  21. 2019Eigenvalue and Generalized Eigenvalue Problems: Tutorialcited by 45
  22. 2023The LLM Surgeoncited by 36
  23. 2019Efficient Subsampled Gauss-Newton and Natural Gradient Methods for Training Neural Networkscited by 32
  24. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 31
  25. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  26. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  27. 2022Indiscriminate Data Poisoning Attacks on Neural Networkscited by 24
  28. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  29. 2024An Improved Empirical Fisher Approximation for Natural Gradient Descentcited by 15
  30. 2024Recent and Upcoming Developments in Randomized Numerical Linear Algebra for Machine Learningcited by 11
  31. 2024Searching for Efficient Linear Layers over a Continuous Space of Structured Matricescited by 11
  32. 2024Trade-Offs of Diagonal Fisher Information Matrix Estimatorscited by 10
  33. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  34. 2021Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networkscited by 7
  35. 2023How to guess a gradientcited by 7
  36. 2023ISAAC Newton: Input-based Approximate Curvature for Newton's Methodcited by 6
  37. 2024Ginger: An Efficient Curvature Approximation with Linear Complexity for General Neural Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2026Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attributioncited by 6
  2. 2025laplax -- Laplace Approximations with JAXcited by 5
  3. 2026Gauss-Newton Unlearning for the LLM Eracited by 4
  4. 2025Distributional Training Data Attribution: What do Influence Functions Sample?cited by 3
  5. 2025Fast Generalized Neural Tangent Kernel Statistics via Trace Estimationcited by 3
  6. 2025Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methodscited by 1
  7. 2026Efficient Bilevel Optimization with KFAC-Based Hypergradientscited by 1
  8. 2026Bayesian Fine-tuning in Projected Subspacescited by 0
  9. 2026Subspace Inference Enables Efficient Active Reward Learning from Preferencescited by 0