mlsanityData to 7 Oct 2026

Scaling Up Influence Functions

arXiv 2112.03052 ↗First posted 6 Dec 2021cs.LG cs.CL cs.CVcited by 130

Not on the map. Why?

Abstract (arXiv)

We address efficient calculation of influence functions for tracking predictions back to the training data. We propose and analyze a new approach to speeding up the inverse Hessian calculation based on Arnoldi iteration. With this improvement, we achieve, to the best of our knowledge, the first successful implementation of influence functions that scales to full-size (language and vision) Transformer models with several hundreds of millions of parameters. We evaluate our approach on image classification and sequence-to-sequence tasks with tens to a hundred of millions of training examples. Our code will be available at https://github.com/google-research/jax-influence.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 6 Dec 2021.

Builds on

28 of 42 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2015Automatic differentiation in machine learning: a surveycited by 846
  8. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  9. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  10. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  11. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
  12. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
16 more
  1. 2014Sketching as a Tool for Numerical Linear Algebracited by 285
  2. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  3. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  4. 2016Understanding Neural Networks through Representation Erasurecited by 223
  5. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  6. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  7. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  8. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  9. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  10. 2018Understanding the Origins of Bias in Word Embeddingscited by 90
  11. 2018Data Dropout: Optimizing Training Data for Convolutional Neural Networkscited by 23
  12. 2016"Influence Sketching": Finding Influential Samples In Large-Scale Regressionscited by 17
  13. 2020Multi-Stage Influence Functioncited by 16
  14. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16
  15. 2018A Fast, Compact, Accurate Model for Language Identification of Codemixed Textcited by 11
  16. 2018Microsoft's Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Datacited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 23; 2024: 34; 2025: 45; 2026: 18.

130 papers.

Sort:

By area:

  1. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  2. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  3. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  4. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 121
  5. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  6. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 88
  7. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 52
  8. 2023Intriguing Properties of Data Attribution on Diffusion Modelscited by 47
  9. 2023Evaluating Data Attribution for Text-to-Image Modelscited by 45
  10. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  11. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearningcited by 44
  12. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 42
38 more
  1. 2024Scalable Influence and Fact Tracing for Large Language Model Pretrainingcited by 40
  2. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 33
  3. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 31
  4. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  5. 2023Theoretical and Practical Perspectives on what Influence Functions Docited by 28
  6. 2022Fair Infinitesimal Jackknife: Mitigating the Influence of Biased Training Data Points Without Refittingcited by 26
  7. 2022Understanding Influence Functions and Datamodels via Harmonic Analysiscited by 23
  8. 2024Data Attribution for Text-to-Image Models by Unlearning Synthesized Imagescited by 22
  9. 2022Towards Tracing Factual Knowledge in Language Models Back to the Training Datacited by 21
  10. 2023Regularizing Second-Order Influences for Continual Learningcited by 21
  11. 2024Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Modelscited by 21
  12. 2024Do Influence Functions Work on Large Language Models?cited by 20
  13. 2022Membership Inference Attack Using Self Influence Functionscited by 19
  14. 2023Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasetscited by 18
  15. 2024textttdattri: A Library for Efficient Data Attributioncited by 18
  16. 2023Natural Example-Based Explainability: a Surveycited by 13
  17. 2022Contrastive Error Attribution for Finetuned Language Modelscited by 12
  18. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  19. 2023A Bayesian Approach To Analysing Training Data Attribution In Deep Learningcited by 11
  20. 2023Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learningcited by 11
  21. 2023Sample based Explanations via Generalized Representerscited by 10
  22. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  23. 2024Efficient Sketches for Training Data Attribution and Studying the Loss Landscapecited by 10
  24. 2024Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Modelscited by 10
  25. 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
  26. 2022Influence Functions for Sequence Tagging Modelscited by 9
  27. 2024Unveiling Privacy, Memorization, and Input Curvature Linkscited by 9
  28. 2025Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learningcited by 9
  29. 2024Revisiting inverse Hessian vector products for calculating influence functionscited by 8
  30. 2025Enhancing Training Data Attribution with Representational Optimizationcited by 8
  31. 2025GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projectioncited by 8
  32. 2023Error Discovery by Clustering Influence Embeddingscited by 7
  33. 2023Deeper Understanding of Black-box Predictions via Generalized Influence Functionscited by 7
  34. 2025IF-GUIDE: Influence Function-Guided Detoxification of LLMscited by 7
  35. 2025Fast Data Attribution for Text-to-Image Modelscited by 7
  36. 2023Efficient Parametric Approximations of Neural Network Function Space Distancecited by 6
  37. 2024Sparse, Efficient and Explainable Data Attribution with DualXDAcited by 6
  38. 2023Gradient-Based Automated Iterative Recovery for Parameter-Efficient Tuningcited by 5

50 most cited of 130 All 130 (JSON)