# Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions

arXiv: https://arxiv.org/abs/2509.23437 · HTML: https://mlsanity.com/paper/2509.23437/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 42 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Builds on (most cited first, 30 of 35)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [Towards Efficient Data Valuation Based on the Shapley Value](https://mlsanity.com/paper/1902.10275/) (2019)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [On the Accuracy of Influence Functions for Measuring Group Effects](https://mlsanity.com/paper/1905.13289/) (2019)
- [Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis](https://mlsanity.com/paper/1806.03884/) (2018)
- [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022)
- [Practical Gauss-Newton Optimisation for Deep Learning](https://mlsanity.com/paper/1706.03662/) (2017)
- [Training Data Influence Analysis and Estimation: A Survey](https://mlsanity.com/paper/2212.04612/) (2022)
- [Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://mlsanity.com/paper/2407.13623/) (2024)
- [Influence Selection for Active Learning](https://mlsanity.com/paper/2108.09331/) (2021)
- [Analytic Insights into Structure and Rank of Neural Network Hessian Maps](https://mlsanity.com/paper/2106.16225/) (2021)
- [Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures](https://mlsanity.com/paper/2311.00636/) (2023)
- [Training Data Attribution via Approximate Unrolled Differentiation](https://mlsanity.com/paper/2405.12186/) (2024)
- [Influence Functions for Scalable Data Attribution in Diffusion Models](https://mlsanity.com/paper/2410.13850/) (2024)
- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [MAGIC: Near-Optimal Data Attribution for Deep Learning](https://mlsanity.com/paper/2504.16430/) (2025)
- [Revisiting the Fragility of Influence Functions](https://mlsanity.com/paper/2303.12922/) (2023)
- [ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure](https://mlsanity.com/paper/2106.02624/) (2021)
- [Modular Block-diagonal Curvature Approximations for Feedforward Architectures](https://mlsanity.com/paper/1902.01813/) (2019)
- [Towards Quantifying the Hessian Structure of Neural Networks](https://mlsanity.com/paper/2505.02809/) (2025)
- [On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning](https://mlsanity.com/paper/2502.01763/) (2025)
- [Block-diagonal Hessian-free Optimization for Training Neural Networks](https://mlsanity.com/paper/1712.07296/) (2017)
- [The Hessian perspective into the Nature of Convolutional Neural Networks](https://mlsanity.com/paper/2305.09088/) (2023)

## Built on by (most cited first, 3 of 3)

- [SOAP-Bubbles: Structured Weight Uncertainty for Neural Networks](https://mlsanity.com/paper/2606.23357/) (2026)
- [Sketching the Readout of Large Language Models for Scalable Data Attribution and Valuation](https://mlsanity.com/paper/2604.16197/) (2026)
- [Quantifying the Agreement Between Data-Influence and Data-Similarity to Understand LLM Behavior](https://mlsanity.com/paper/2606.23591/) (2026)

## Abstract (arXiv)

Influence functions offer a principled way to trace model predictions back to training data, but their use in deep learning is hampered by the need to invert a large, ill-conditioned Hessian matrix. Approximations such as Generalised Gauss-Newton (GGN) and Kronecker-Factored Approximate Curvature (K-FAC) have been proposed to make influence computation tractable, yet it remains unclear how the departure from exactness impacts data attribution performance. Critically, given the restricted regime in which influence functions are derived, it is not necessarily clear better Hessian approximations should even lead to better data attribution performance. In this paper, we investigate the effect of Hessian approximation quality on influence-function attributions in a controlled classification setting. Our experiments show that better Hessian approximations consistently yield better influence score quality, offering justification for recent research efforts towards that end. We further decompose the approximation steps for recent Hessian approximation methods and evaluate each step's influence on attribution accuracy. Notably, the mismatch between K-FAC eigenvalues and GGN/EK-FAC eigenvalues accounts for the majority of the error and influence loss. These findings highlight which approximations are most critical, guiding future efforts to balance computational tractability and attribution accuracy.