mlsanityData to 14 Oct 2026

Training Data Attribution via Approximate Unrolled Differentiation

arXiv 2405.12186 ↗First posted 20 May 2024cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Many training data attribution (TDA) methods aim to estimate how a model's behavior would change if one or more data points were removed from the training set. Methods based on implicit differentiation, such as influence functions, can be made computationally efficient, but fail to account for underspecification, the implicit bias of the optimization algorithm, or multi-stage training pipelines. By contrast, methods based on unrolling address these issues but face scalability challenges. In this work, we connect the implicit-differentiation-based and unrolling-based approaches and combine their benefits by introducing Source, an approximate unrolling-based TDA method that is computed using an influence-function-like formula. While being computationally efficient compared to unrolling-based approaches, Source is suitable in cases where implicit-differentiation-based approaches struggle, such as in non-converged models and multi-stage training pipelines. Empirically, Source outperforms existing TDA techniques in counterfactual prediction, especially in settings where implicit-differentiation-based approaches fall short.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 59 of its 94 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

59 of 94 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2016Wide Residual Networkscited by 3,070
  9. 2017Billion-scale similarity search with GPUscited by 2,497
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  11. 2020In Search of Lost Domain Generalizationcited by 1,054
  12. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
38 more
  1. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  2. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  3. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  4. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  6. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  7. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  8. 2014New insights and perspectives on the natural gradient methodcited by 443
  9. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  10. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  11. 2019Towards Efficient Data Valuation Based on the Shapley Valuecited by 281
  12. 2015Domain Generalization for Object Recognition with Multi-task Autoencoderscited by 253
  13. 2020Influence Functions in Deep Learning Are Fragilecited by 252
  14. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  15. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  16. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 165
  17. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  18. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  19. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  20. 2022If Influence Functions are the Answer, Then What is the Question?cited by 144
  21. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  22. 2022Training Data Influence Analysis and Estimation: A Surveycited by 135
  23. 2021Scaling Up Influence Functionscited by 132
  24. 2022Data Banzhaf: A Robust Data Valuation Framework for Machine Learningcited by 132
  25. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
  26. 2021Influence Selection for Active Learningcited by 84
  27. 2020Subpopulation Data Poisoning Attackscited by 70
  28. 2019Data Cleansing for Models Trained with SGDcited by 67
  29. 2020Influence Function based Data Poisoning Attacks to Top-N Recommender Systemscited by 63
  30. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 53
  31. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  32. 2018Interpreting Black Box Predictions using Fisher Kernelscited by 48
  33. 2023Intriguing Properties of Data Attribution on Diffusion Modelscited by 47
  34. 2023The Journey, Not the Destination: How Data Guides Diffusion Modelscited by 43
  35. 2021Revisiting Methods for Finding Influential Examplescited by 39
  36. 2022First is Better Than Last for Language Data Influencecited by 34
  37. 2023Theoretical and Practical Perspectives on what Influence Functions Docited by 29
  38. 2020Explaining and Improving Model Behavior with k Nearest Neighbor Representationscited by 27

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 17; 2026: 8.

34 papers.

Sort:

By area:

  1. 2024Procedural Knowledge in Pretraining Drives Reasoning in Large Language Modelscited by 41
  2. 2024Capturing the Temporal Dependence of Training Data Influencecited by 35
  3. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 33
  4. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 32
  5. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  6. 2025DataMIL: Selecting Data for Robot Imitation Learning with Datamodelscited by 28
  7. 2025MAGIC: Near-Optimal Data Attribution for Deep Learningcited by 22
  8. 2024Attribute-to-Delete: Machine Unlearning via Datamodel Matchingcited by 18
  9. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  10. 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
  11. 2024JacNet: Learning Functions with Structured Jacobianscited by 7
  12. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyondcited by 6
22 more
  1. 2026Infusion: Shaping Model Behavior by Editing Training Data via Influence Functionscited by 5
  2. 2025Efficient Multi-Agent System Training with Data Influence-Oriented Tree Searchcited by 4
  3. 2024Towards User-Focused Research in Training Data Attribution for Human-Centered Explainable AIcited by 3
  4. 2025Unraveling Indirect In-Context Learning Using Influence Functionscited by 3
  5. 2025Distributional Training Data Attribution: What do Influence Functions Sample?cited by 3
  6. 2025Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functionscited by 3
  7. 2025Train on Validation (ToV): Fast data selection with applications to fine-tuningcited by 3
  8. 2025Z0-Inf: Zeroth Order Approximation for Data Influencecited by 3
  9. 2026In-Run Data Shapley for Adam Optimizercited by 3
  10. 2026Bergson: An Open Source Library for Data Attributioncited by 3
  11. 2024Quanda: An Interpretability Toolkit for Training Data Attribution Evaluation and Beyondcited by 2
  12. 2025Layer-Aware Influence for Online Data Valuation Estimationcited by 2
  13. 2026Variance Reduction for Expectations with Diffusion Teacherscited by 2
  14. 2025Small-to-Large Generalization: Data Influences Models Consistently Across Scalecited by 1
  15. 2025Daunce: Data Attribution through Uncertainty Estimationcited by 1
  16. 2026A Bayesian Information-Theoretic Approach to Data Attributioncited by 1
  17. 2026What's a Credit Worth? A Market Framework for Attribution-Aware Compensation in Generative Musiccited by 1
  18. 2025Scalable Multi-Stage Influence Function for Large Language Models via Eigenvalue-Corrected Kronecker-Factored Parameterizationcited by 0
  19. 2025Exploring Training Data Attribution under Limited Access Constraintscited by 0
  20. 2025What Is The Performance Ceiling of My Classifier? Utilizing Category-Wise Influence Functions for Pareto Frontier Analysiscited by 0
  21. 2026Gradient Atoms: Unsupervised Discovery, Attribution and Steering of Model Behaviors via Sparse Decomposition of Training Gradientscited by 0
  22. 2026Validity Threats for Foundation Model Researchcited by 0