mlsanityData to 7 Oct 2026

Better Training Data Attribution via Better Inverse Hessian-Vector Products

arXiv 2507.14740 ↗First posted 19 Jul 2025cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

Training data attribution (TDA) provides insights into which training data is responsible for a learned model behavior. Gradient-based TDA methods such as influence functions and unrolled differentiation both involve a computation that resembles an inverse Hessian-vector product (iHVP), which is difficult to approximate efficiently. We introduce an algorithm (ASTRA) which uses the EKFAC-preconditioner on Neumann series iterations to arrive at an accurate iHVP approximation for TDA. ASTRA is easy to tune, requires fewer iterations than Neumann series iterations, and is more accurate than EKFAC-based approximations. Using ASTRA, we show that improving the accuracy of the iHVP approximation can significantly improve TDA performance.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 65 of its 101 references that are in the map. First posted on arXiv 19 Jul 2025.

Builds on

65 of 101 references.

References · 65

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  10. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  11. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  12. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
38 more
  1. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  2. 2014New insights and perspectives on the natural gradient methodcited by 435
  3. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  4. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  5. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  6. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  7. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  8. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  9. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  10. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  11. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  12. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  13. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 146
  14. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  15. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  16. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  17. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  18. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  19. 2021Scaling Up Influence Functionscited by 130
  20. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 121
  21. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  22. 2018Understanding the Origins of Bias in Word Embeddingscited by 90
  23. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 88
  24. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  25. 2019On Second-Order Group Influence Functions for Black-Box Predictionscited by 78
  26. 2014A Stochastic Quasi-Newton Method for Large-Scale Optimizationcited by 68
  27. 2019Data Cleansing for Models Trained with SGDcited by 67
  28. 2021On Memorization in Probabilistic Deep Generative Modelscited by 60
  29. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 52
  30. 2021Interactive Label Cleaning with Example-based Explanationscited by 48
  31. 2021Revisiting Methods for Finding Influential Examplescited by 39
  32. 2021Combining Feature and Instance Attribution to Detect Artifactscited by 38
  33. 2024Capturing the Temporal Dependence of Training Data Influencecited by 34
  34. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 33
  35. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 31
  36. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 31
  37. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  38. 2023Theoretical and Practical Perspectives on what Influence Functions Docited by 28

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2025Bayesian Influence Functions for Hessian-Free Data Attributioncited by 12
  2. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 9
  3. 2025Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functionscited by 2
  4. 2026LoRIF: Low-Rank Influence Functions for Scalable Training Data Attributioncited by 2
  5. 2026Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamicscited by 2
  6. 2026A Unified Theory of Random Projection for Influence Functionscited by 1
  7. 2026Interaction-Aware Influence Functions for Group Attributioncited by 1
  8. 2026On the Fragility of Data Attribution When Learning Is Distributedcited by 0
  9. 2026Tracing sources of epistemic uncertainty in deep learning predictions: homo- and hetero-scedastic linearized estimatorscited by 0
  10. 2026Which Influence Are We Estimating? The Role of Counterfactual Specifications in Data Attributioncited by 0