# Revisit, Extend, and Enhance Hessian-Free Influence Functions

arXiv: https://arxiv.org/abs/2405.17490 · HTML: https://mlsanity.com/paper/2405.17490/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2024.

## Built on by (most cited first, 9 of 9)

- [WAPITI: A Watermark for Finetuned Open-Source LLMs](https://mlsanity.com/paper/2410.06467/) (2024)
- [Step-resolved data attribution for looped transformers](https://mlsanity.com/paper/2602.10097/) (2026)
- [DMin: Scalable Training Data Influence Estimation for Diffusion Models](https://mlsanity.com/paper/2412.08637/) (2024)
- [Understanding Data Influence with Differential Approximation](https://mlsanity.com/paper/2508.14648/) (2025)
- [Layer-Aware Influence for Online Data Valuation Estimation](https://mlsanity.com/paper/2510.16007/) (2025)
- [Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models Via Layer Gradient Analysis](https://mlsanity.com/paper/2602.20207/) (2026)
- [An Isotropic Approach to Efficient Uncertainty Quantification with Gradient Norms](https://mlsanity.com/paper/2603.29466/) (2026)
- [From Detrimental to Beneficial: Dynamic Influence-based Valuation and Editing](https://mlsanity.com/paper/2608.22522/) (2026)
- [Not All Rollouts Are Worth Learning: On Trajectory Valuation for Post-Training Reinforcement Learning](https://mlsanity.com/paper/2609.35072/) (2026)

## Abstract (arXiv)

Influence functions serve as crucial tools for assessing sample influence in model interpretation, subset training set selection, noisy label detection, and more. By employing the first-order Taylor extension, influence functions can estimate sample influence without the need for expensive model retraining. However, applying influence functions directly to deep models presents challenges, primarily due to the non-convex nature of the loss function and the large size of model parameters. This difficulty not only makes computing the inverse of the Hessian matrix costly but also renders it non-existent in some cases. Various approaches, including matrix decomposition, have been explored to expedite and approximate the inversion of the Hessian matrix, with the aim of making influence functions applicable to deep models. In this paper, we revisit a specific, albeit naive, yet effective approximation method known as TracIn. This method substitutes the inverse of the Hessian matrix with an identity matrix. We provide deeper insights into why this simple approximation method performs well. Furthermore, we extend its applications beyond measuring model utility to include considerations of fairness and robustness. Finally, we enhance TracIn through an ensemble strategy. To validate its effectiveness, we conduct experiments on synthetic data and extensive evaluations on noisy label detection, sample selection for large language model fine-tuning, and defense against adversarial attacks.