mlsanityData to 7 Oct 2026

Training Data Influence Analysis and Estimation: A Survey

arXiv 2212.04612 ↗First posted 9 Dec 2022cs.LGcited by 134

Not on the map. Why?

Abstract (arXiv)

Good models require good training data. For overparameterized deep models, the causal relationship between training data and model predictions is increasingly opaque and poorly understood. Influence analysis partially demystifies training's underlying interactions by quantifying the amount each training instance alters the final model. Measuring the training data's influence exactly can be provably hard in the worst case; this has led to the development and use of influence estimators, which only approximate the true influence. This paper provides the first comprehensive survey of training data influence analysis and estimation. We begin by formalizing the various, and in places orthogonal, definitions of training data influence. We then organize state-of-the-art influence analysis methods into a taxonomy; we describe each of these methods in detail and compare their underlying assumptions, asymptotic complexities, and overall strengths and weaknesses. Finally, we propose future research directions to make influence analysis more useful in practice as well as more theoretically and empirically sound. A curated, up-to-date list of resources related to influence analysis is available at https://github.com/ZaydH/influence_analysis_papers.

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). Builds on 105 of its 207 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

105 of 207 references.

References · 105

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2021Convex optimizationcited by 2,479
  7. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  8. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  12. 2017A Closer Look at Memorization in Deep Networkscited by 936
38 more
  1. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  2. 2016The Mythos of Model Interpretabilitycited by 716
  3. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  4. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  5. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550
  6. 2020A Survey of Deep Active Learningcited by 547
  7. 2019Benign Overfitting in Linear Regressioncited by 544
  8. 2017Large Batch Training of Convolutional Networkscited by 529
  9. 2019Certified Data Removal from Machine Learning Modelscited by 512
  10. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  11. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  12. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  13. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  14. 2020Backdoor Learning: A Surveycited by 412
  15. 2019The many Shapley values for model explanationcited by 341
  16. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  17. 2017Certified Defenses for Data Poisoning Attackscited by 288
  18. 2014Sketching as a Tool for Numerical Linear Algebracited by 285
  19. 2019Towards Efficient Data Valuation Based on the Shapley Valuecited by 279
  20. 2019Measuring Bias in Contextualized Word Representationscited by 275
  21. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 259
  22. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  23. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
  24. 2020Coresets via Bilevel Optimization for Continual Learning and Streamingcited by 191
  25. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  26. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  27. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 162
  28. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  29. 2019Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithmscited by 147
  30. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 146
  31. 2019The Shapley Taylor Interaction Indexcited by 143
  32. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  33. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  34. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  35. 2021Scaling Up Influence Functionscited by 130
  36. 2022Data Banzhaf: A Robust Data Valuation Framework for Machine Learningcited by 126
  37. 2020Machine Unlearning for Random Forestscited by 125
  38. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 34; 2025: 53; 2026: 32.

134 papers.

Sort:

By area:

  1. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  2. 2023Intriguing Properties of Data Attribution on Diffusion Modelscited by 47
  3. 2024Incremental Residual Concept Bottleneck Modelscited by 43
  4. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 42
  5. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 33
  6. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 31
  7. 2024ICONS: Influence Consensus for Vision-Language Data Selectioncited by 28
  8. 2025DataMIL: Selecting Data for Robot Imitation Learning with Datamodelscited by 25
  9. 2023Training Data Attribution for Diffusion Modelscited by 24
  10. 2025Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilitiescited by 22
  11. 2024Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Modelscited by 21
  12. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 21
38 more
  1. 2025MAGIC: Near-Optimal Data Attribution for Deep Learningcited by 20
  2. 2024Attribute-to-Delete: Machine Unlearning via Datamodel Matchingcited by 18
  3. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 17
  4. 2024The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passescited by 16
  5. 2023Large Language Models Are Better Adversaries: Exploring Generative Clean-Label Backdoor Attacks Against Text Classifierscited by 14
  6. 2024Remaining-data-free Machine Unlearning by Suppressing Sample Contributioncited by 13
  7. 2024Fairness Without Harm: An Influence-Guided Active Sampling Approachcited by 12
  8. 2023A Bayesian Approach To Analysing Training Data Attribution In Deep Learningcited by 11
  9. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
  10. 2023Sample based Explanations via Generalized Representerscited by 10
  11. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  12. 2024Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Modelscited by 10
  13. 2024Generative Active Learning for Long-tailed Instance Segmentationcited by 10
  14. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  15. 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
  16. 2024Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspectivecited by 9
  17. 2024In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Modelscited by 9
  18. 2025GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projectioncited by 8
  19. 2023Towards Algorithmic Fairness by means of Instance-level Data Re-weighting based on Shapley Valuescited by 7
  20. 2025Fairshare Data Pricing via Data Valuation for Large Language Modelscited by 7
  21. 2025Beyond Similarity: A Gradient-based Graph Method for Instruction Tuning Data Selectioncited by 7
  22. 2023Data Optimization in Deep Learning: A Surveycited by 6
  23. 2024Task-oriented Time Series Imputation Evaluation via Generalized Representerscited by 6
  24. 2025Influence Guided Context Selection for Effective Retrieval-Augmented Generationcited by 6
  25. 2026Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attributioncited by 6
  26. 2023A Simple and Efficient Baseline for Data Attribution on Imagescited by 5
  27. 2024Improve Knowledge Distillation via Label Revision and Data Selectioncited by 5
  28. 2024Efficient Ensembles Improve Training Data Attributioncited by 5
  29. 2024Anytime-Valid Inference for Double/Debiased Machine Learning of Causal Parameterscited by 5
  30. 2024A Versatile Influence Function for Data Attribution with Non-Decomposable Losscited by 5
  31. 2025DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Modelscited by 5
  32. 2025First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimationcited by 5
  33. 2026Step-resolved data attribution for looped transformerscited by 5
  34. 2024LightHouse: A Survey of AGI Hallucinationcited by 4
  35. 2024Understanding Multimodal Deep Neural Networks: A Concept Selection Viewcited by 4
  36. 2024On the Inflation of KNN-Shapley Valuecited by 4
  37. 2024Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methodscited by 4
  38. 2025Towards Robust Influence Functions with Flat Validation Minimacited by 4

50 most cited of 134 All 134 (JSON)