mlsanityData to 7 Oct 2026

DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models

arXiv 2310.00902 ↗First posted 2 Oct 2023cs.LG stat.MLcited by 121

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Quantifying the impact of training data points is crucial for understanding the outputs of machine learning models and for improving the transparency of the AI pipeline. The influence function is a principled and popular data attribution method, but its computational cost often makes it challenging to use. This issue becomes more pronounced in the setting of large language models and text-to-image models. In this work, we propose DataInf, an efficient influence approximation method that is practical for large-scale generative AI models. Leveraging an easy-to-compute closed-form expression, DataInf outperforms existing influence computation algorithms in terms of computational and memory efficiency. Our theoretical analysis shows that DataInf is particularly well-suited for parameter-efficient fine-tuning techniques such as LoRA. Through systematic empirical evaluations, we show that DataInf accurately approximates influence scores and is orders of magnitude faster than existing methods. In applications to RoBERTa-large, Llama-2-13B-chat, and stable-diffusion-v1.5 models, DataInf effectively identifies the most influential fine-tuning examples better than other approximate influence scores. Moreover, it can help to identify which data points are mislabeled.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 36 of its 58 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

36 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  2. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  11. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  12. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
22 more
  1. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  2. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  3. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  4. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  5. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  6. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  7. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 162
  8. 2019Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithmscited by 147
  9. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  10. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  11. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  12. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  13. 2021Scaling Up Influence Functionscited by 130
  14. 2022Data Banzhaf: A Robust Data Valuation Framework for Machine Learningcited by 126
  15. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  16. 2019Data Valuation using Reinforcement Learningcited by 115
  17. 2021Input Similarity from the Neural Network Perspectivecited by 78
  18. 2023OpenDataVal: a Unified Benchmark for Data Valuationcited by 55
  19. 2019Repairing without Retraining: Avoiding Disparate Impact with Counterfactual Distributionscited by 48
  20. 2023Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Valuecited by 38
  21. 2022Information Theory with Kernel Methodscited by 37
  22. 2023Data Valuation Without Training of a Modelcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 38; 2025: 37; 2026: 43.

121 papers.

Sort:

By area:

  1. 2024ContextCite: Attributing Model Generation to Contextcited by 93
  2. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 88
  3. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  4. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  5. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  6. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 33
  7. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 31
  8. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  9. 2024Editable Concept Bottleneck Modelscited by 22
  10. 2024Data Attribution for Text-to-Image Models by Unlearning Synthesized Imagescited by 22
  11. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  12. 2024Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attributioncited by 21
38 more
  1. 2024Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Modelscited by 21
  2. 2023Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrationscited by 20
  3. 2024Do Influence Functions Work on Large Language Models?cited by 20
  4. 2024Hippocrates: An Open-Source Framework for Advancing Large Language Models in Healthcarecited by 15
  5. 2024Helpful or Harmful Data? Fine-tuning-free Shapley Attribution for Explaining Language Model Predictionscited by 13
  6. 2025LEAD: Iterative Data Selection for Efficient LLM Instruction Tuningcited by 13
  7. 2025LayerIF: Estimating Layer Quality for Large Language Models using Influence Functionscited by 12
  8. 2025Understanding Impact of Human Feedback via Influence Functionscited by 11
  9. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  10. 2024Outlier Gradient Analysis: Efficiently Identifying Detrimental Training Samples for Deep Learning Modelscited by 10
  11. 2024Scaling Laws for the Value of Individual Data Points in Machine Learningcited by 10
  12. 2025Better Training Data Attribution via Better Inverse Hessian-Vector Productscited by 10
  13. 2024Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspectivecited by 9
  14. 2024More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthinesscited by 9
  15. 2024In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Modelscited by 9
  16. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 9
  17. 2025Enhancing Training Data Attribution with Representational Optimizationcited by 8
  18. 2025GraSS: Scalable Data Attribution with Gradient Sparsification and Sparse Projectioncited by 8
  19. 2023Deeper Understanding of Black-box Predictions via Generalized Influence Functionscited by 7
  20. 2024Dissecting Representation Misalignment in Contrastive Learning via Influence Functioncited by 7
  21. 2025Fairshare Data Pricing via Data Valuation for Large Language Modelscited by 7
  22. 2025Augmenting Perceptual Super-Resolution via Image Quality Predictorscited by 7
  23. 2025IF-GUIDE: Influence Function-Guided Detoxification of LLMscited by 7
  24. 2024TimeInf: Time Series Data Contribution via Influence Functionscited by 6
  25. 2025Skill-Targeted Adaptive Trainingcited by 6
  26. 2024LLM Attributor: Interactive Visual Attribution for LLM Generationcited by 5
  27. 2024DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learningcited by 5
  28. 2024Efficient Ensembles Improve Training Data Attributioncited by 5
  29. 2024A Versatile Influence Function for Data Attribution with Non-Decomposable Losscited by 5
  30. 2025DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Modelscited by 5
  31. 2025First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimationcited by 5
  32. 2025To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysiscited by 5
  33. 2024Enhancing Data Quality in Federated Fine-Tuning of Foundation Modelscited by 4
  34. 2024Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methodscited by 4
  35. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  36. 2025Towards Robust Influence Functions with Flat Validation Minimacited by 4
  37. 2025AutoMixer: Checkpoint Artifacts as Automatic Data Mixerscited by 4
  38. 2025CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamicscited by 4

50 most cited of 121 All 121 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DeepSN: A Sheaf Neural Framework for Influence Maximization
  2. 2025Data-Efficient RLVR via Off-Policy Influence Guidance
  3. 2025Nonparametric Data Attribution for Diffusion Models
  4. 2025REM: A Scalable Reinforced Multi-Expert Framework for Multiplex Influence Maximization
  5. 2026Rethinking Learning-Based Influence Maximization: Simple Neural Surrogates and Native Discrete Search
  6. 2026ATHENA: Accelerated Multi-Task Heterogeneous Influence Functions for Robot Data Curation
  7. 2026LoRA-Diffusion: Parameter-Efficient Fine-Tuning via Low-Rank Trajectory Decomposition
  8. 2024Correcting Large Language Model Behavior via Influence Function