mlsanityData to 7 Oct 2026

Data Shapley in One Training Run

arXiv 2406.11011 ↗First posted 16 Jun 2024cs.LG cs.CL stat.MLcited by 76

Not on the map. Why?

Abstract (arXiv)

Data Shapley provides a principled framework for attributing data's contribution within machine learning contexts. However, existing approaches require re-training models on different data subsets, which is computationally intensive, foreclosing their application to large-scale models. Furthermore, they produce the same attribution score for any models produced by running the learning algorithm, meaning they cannot perform targeted attribution towards a specific model obtained from a single run of the algorithm. This paper introduces In-Run Data Shapley, which addresses these limitations by offering scalable data attribution for a target model of interest. In its most efficient implementation, our technique incurs negligible additional runtime compared to standard model training. This dramatic efficiency improvement makes it possible to perform data attribution for the foundation model pretraining stage for the first time. We present several case studies that offer fresh insights into pretraining data's contribution and discuss their implications for copyright in generative AI and pretraining data curation.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 53 of its 71 references that are in the map. First posted on arXiv 16 Jun 2024.

Builds on

53 of 71 references.

References · 53

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  7. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  8. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  9. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  10. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  11. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  12. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
38 more
  1. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  2. 2019Towards Efficient Data Valuation Based on the Shapley Valuecited by 279
  3. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  4. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  5. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  6. 2021Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learningcited by 162
  7. 2019Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithmscited by 147
  8. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  9. 2020A Principled Approach to Data Valuation for Federated Learningcited by 127
  10. 2022Data Banzhaf: A Robust Data Valuation Framework for Machine Learningcited by 126
  11. 2023Foundation Models and Fair Usecited by 112
  12. 2021Sampling Permutations for Shapley Value Estimationcited by 101
  13. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 88
  14. 2020A Distributional Framework for Data Valuationcited by 79
  15. 2023LAVA: Data Valuation without Pre-Specified Learning Algorithmscited by 69
  16. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  17. 2023OpenDataVal: a Unified Benchmark for Data Valuationcited by 55
  18. 2022Measuring the Effect of Training Data on Deep Learning Predictions via Randomized Experimentscited by 50
  19. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  20. 2019Emergent properties of the local geometry of neural loss landscapescited by 46
  21. 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 46
  22. 2021Revisiting Methods for Finding Influential Examplescited by 39
  23. 2020Efficient computation and analysis of distributional Shapley valuescited by 38
  24. 2023Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Valuecited by 38
  25. 2023Data Valuation Without Training of a Modelcited by 35
  26. 2020A Multilinear Sampling Algorithm to Estimate Shapley Valuescited by 33
  27. 2024Rethinking Data Shapley for Data Selection Tasks: Misleads and Meritscited by 29
  28. 2022Understanding Influence Functions and Datamodels via Harmonic Analysiscited by 23
  29. 2024Stochastic Amortization: A Unified Approach to Accelerate Feature and Data Attributioncited by 21
  30. 2021Incentivizing Collaboration in Machine Learning via Synthetic Data Rewardscited by 20
  31. 2022Fundamentals of Task-Agnostic Data Valuationcited by 19
  32. 2023Threshold KNN-Shapley: A Linear-Time and Privacy-Friendly Approach to Data Valuationcited by 19
  33. 2024Efficient Data Shapley for Weighted Nearest Neighbor Algorithmscited by 18
  34. 2024An Economic Solution to Copyright Challenges of Generative AIcited by 18
  35. 2019Efficient Per-Example Gradient Computations in Convolutional Neural Networkscited by 17
  36. 2019Estimation of the Shapley value by ergodic samplingcited by 15
  37. 2023A Note on "Towards Efficient Data Valuation Based on the Shapley Value''cited by 11
  38. 2023A Note on "Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms"cited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 33; 2026: 39.

76 papers.

Sort:

By area:

  1. 2024Data Selection via Optimal Control for Language Modelscited by 38
  2. 2024Capturing the Temporal Dependence of Training Data Influencecited by 34
  3. 2025Regression-adjusted Monte Carlo Estimators for Shapley Values and Probabilistic Valuescited by 19
  4. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 17
  5. 2025DataRater: Meta-Learned Dataset Curationcited by 13
  6. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11
  7. 2026Shaping capabilities with token-level data filteringcited by 11
  8. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  9. 2025Fairshare Data Pricing via Data Valuation for Large Language Modelscited by 7
  10. 2025Fast Data Attribution for Text-to-Image Modelscited by 7
  11. 2026MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Mergingcited by 7
  12. 2025Train Once, Answer All: Many Pretraining Experiments for the Cost of Onecited by 6
38 more
  1. 2025CAMs as Shapley Value-based Explainerscited by 5
  2. 2025DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Modelscited by 5
  3. 2024Targeted synthetic data generation for tabular data via hardness characterizationcited by 4
  4. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  5. 2025Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retrainingcited by 4
  6. 2025Trade in Minutes! Rationality-Driven Agentic System for Quantitative Financial Tradingcited by 4
  7. 2026Motion Attribution for Video Generationcited by 4
  8. 2026How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelinescited by 4
  9. 2025Concept-TRAK: Understanding how diffusion models learn concepts through concept-level attributioncited by 3
  10. 2025MaxShapley: Towards Incentive-compatible Generative Search with Fair Context Attributioncited by 3
  11. 2026A Sustainable AI Economy Needs Data Deals That Work for Generatorscited by 3
  12. 2026OATS: Online Data Augmentation for Time Series Foundation Modelscited by 3
  13. 2026In-Run Data Shapley for Adam Optimizercited by 3
  14. 2025Do Data Valuations Make Good Data Prices?cited by 2
  15. 2025DICE: Data Influence Cascade in Decentralized Learningcited by 2
  16. 2025Newfluence: Boosting Model interpretability and Understanding in High Dimensionscited by 2
  17. 2025Z0-Inf: Zeroth Order Approximation for Data Influencecited by 2
  18. 2025Layer-Aware Influence for Online Data Valuation Estimationcited by 2
  19. 2026GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selectioncited by 2
  20. 2026ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Lawscited by 2
  21. 2024LossVal: Efficient Data Valuation for Neural Networkscited by 1
  22. 2025Unifying and Optimizing Data Values for Selection via Sequential Decision-Makingcited by 1
  23. 2025KAIROS: Scalable Model-Agnostic Data Valuationcited by 1
  24. 2025For-Value: Efficient Forward-Only Data Valuation for finetuning LLMs and VLMscited by 1
  25. 2025AssayMatch: Learning to Select Data for Molecular Activity Modelscited by 1
  26. 2025Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimizationcited by 1
  27. 2026Imperfect Influence, Preserved Rankings: A Theory of TRAK for Data Attributioncited by 1
  28. 2026Local Shapley: Model-Induced Locality and Optimal Reuse in Data Valuationcited by 1
  29. 2026Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuningcited by 1
  30. 2026In-Context Credit Assignment via the Corecited by 1
  31. 2026AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Trainingcited by 1
  32. 2026Measuring Task-Agnostic Training Data Influence Across Language Model Pretrainingcited by 1
  33. 2025Chunked Data Shapley: A Scalable Dataset Quality Assessment for Machine Learningcited by 0
  34. 2025Uncovering Scaling Laws for Large Language Models via Inverse Problemscited by 0
  35. 2025What Is The Performance Ceiling of My Classifier? Utilizing Category-Wise Influence Functions for Pareto Frontier Analysiscited by 0
  36. 2025f-INE: A Hypothesis Testing Framework for Estimating Influence under Training Randomnesscited by 0
  37. 2025Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniquenesscited by 0
  38. 2025Geometric Data Valuation via Leverage Scorescited by 0

50 most cited of 76 All 76 (JSON)