mlsanityData to 7 Oct 2026

If Influence Functions are the Answer, Then What is the Question?

arXiv 2209.05364 ↗First posted 12 Sep 2022cs.LG stat.MLcited by 142

Not on the map. Why?

Abstract (arXiv)

Influence functions efficiently estimate the effect of removing a single training data point on a model's learned parameters. While influence estimates align well with leave-one-out retraining for linear models, recent works have shown this alignment is often poor in neural networks. In this work, we investigate the specific factors that cause this discrepancy by decomposing it into five separate terms. We study the contributions of each term on a variety of architectures and datasets and how they vary with factors such as network width and training time. While practical influence function estimates may be a poor match to leave-one-out retraining for nonlinear networks, we show they are often a good approximation to a different object we term the proximal Bregman response function (PBRF). Since the PBRF can still be used to answer many of the questions motivating influence functions, such as identifying influential or mislabeled examples, our results suggest that current algorithms for influence function estimation give more informative results than previous error analyses would suggest.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Builds on 34 of its 74 references that are in the map. First posted on arXiv 12 Sep 2022.

Builds on

34 of 74 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2016Wide Residual Networkscited by 3,057
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  8. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  9. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  10. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  12. 2019Benign Overfitting in Linear Regressioncited by 544
22 more
  1. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  2. 2017Interpretation of Neural Networks is Fragilecited by 443
  3. 2014New insights and perspectives on the natural gradient methodcited by 435
  4. 2020Approximate Data Deletion from Machine Learning Modelscited by 250
  5. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  6. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  7. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 146
  8. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  9. 2021Scaling Up Influence Functionscited by 130
  10. 2018Stronger Data Poisoning Attacks Break Data Sanitization Defensescited by 125
  11. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  12. 2018Understanding the Origins of Bias in Word Embeddingscited by 90
  13. 2019On Second-Order Group Influence Functions for Black-Box Predictionscited by 78
  14. 2021Input Similarity from the Neural Network Perspectivecited by 78
  15. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  16. 2018Interpreting Black Box Predictions using Fisher Kernelscited by 48
  17. 2021Interactive Label Cleaning with Example-based Explanationscited by 48
  18. 2021Revisiting Methods for Finding Influential Examplescited by 39
  19. 2020When Does Preconditioning Help or Hurt Generalization?cited by 30
  20. 2019Can You Trust This Prediction? Auditing Pointwise Reliability After Learningcited by 28
  21. 2022Understanding Rare Spurious Correlations in Neural Networkscited by 27
  22. 2016Second-Order Stochastic Optimization for Machine Learning in Linear Timecited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 25; 2024: 45; 2025: 44; 2026: 25.

142 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  3. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  4. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  5. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  6. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 121
  7. 2024Guardrail Baselines for Unlearning in LLMscited by 90
  8. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 88
  9. 2023Data Distillation: A Surveycited by 86
  10. 2024Data Shapley in One Training Runcited by 76
  11. 2023Fast Model Debias with Machine Unlearningcited by 70
  12. 2024Are we making progress in unlearning? Findings from the first NeurIPS unlearning competitioncited by 48
38 more
  1. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  2. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 42
  3. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  4. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 33
  5. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 31
  6. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 31
  7. 2023Theoretical and Practical Perspectives on what Influence Functions Docited by 28
  8. 2024Reconstruction Attacks on Machine Unlearning: Simple Models are Vulnerablecited by 24
  9. 2022Understanding Influence Functions and Datamodels via Harmonic Analysiscited by 23
  10. 2024AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attributioncited by 23
  11. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 21
  12. 2023Citation: A Key to Building Responsible and Accountable Large Language Modelscited by 20
  13. 2024Causal Estimation of Memorisation Profilescited by 20
  14. 2024Do Influence Functions Work on Large Language Models?cited by 20
  15. 2025MAGIC: Near-Optimal Data Attribution for Deep Learningcited by 20
  16. 2024Token-wise Influential Training Data Retrieval for Large Language Modelscited by 19
  17. 2022Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curvecited by 18
  18. 2024textttdattri: A Library for Efficient Data Attributioncited by 18
  19. 2024How Much Can We Forget about Data Contamination?cited by 14
  20. 2023Natural Example-Based Explainability: a Surveycited by 13
  21. 2024Remaining-data-free Machine Unlearning by Suppressing Sample Contributioncited by 13
  22. 2024Montessori-Instruct: Generate Influential Training Data Tailored for Student Learningcited by 12
  23. 2023A Bayesian Approach To Analysing Training Data Attribution In Deep Learningcited by 11
  24. 2025Distillation Robustifies Unlearningcited by 11
  25. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  26. 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
  27. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  28. 2024Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspectivecited by 9
  29. 2024Unveiling Privacy, Memorization, and Input Curvature Linkscited by 9
  30. 2024Revisiting inverse Hessian vector products for calculating influence functionscited by 8
  31. 2025Enhancing Training Data Attribution with Representational Optimizationcited by 8
  32. 2023Towards Algorithmic Fairness by means of Instance-level Data Re-weighting based on Shapley Valuescited by 7
  33. 2023Deeper Understanding of Black-box Predictions via Generalized Influence Functionscited by 7
  34. 2023Efficient Parametric Approximations of Neural Network Function Space Distancecited by 6
  35. 2023Understanding Deep Gradient Leakage via Inversion Influence Functionscited by 6
  36. 2024Sparse, Efficient and Explainable Data Attribution with DualXDAcited by 6
  37. 2024Enhancing Training Data Attribution for Large Language Models with Fitting Error Considerationcited by 6
  38. 2025TAPE: Tailored Posterior Difference for Auditing of Machine Unlearningcited by 6

50 most cited of 142 All 142 (JSON)