# Data Shapley in One Training Run

arXiv: https://arxiv.org/abs/2406.11011 · HTML: https://mlsanity.com/paper/2406.11011/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 71 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Builds on (most cited first, 30 of 53)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [Towards Efficient Data Valuation Based on the Shapley Value](https://mlsanity.com/paper/1902.10275/) (2019)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning](https://mlsanity.com/paper/2110.14049/) (2021)
- [Efficient Task-Specific Data Valuation for Nearest Neighbor Algorithms](https://mlsanity.com/paper/1908.08619/) (2019)
- [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022)
- [Data Banzhaf: A Robust Data Valuation Framework for Machine Learning](https://mlsanity.com/paper/2205.15466/) (2022)
- [A Principled Approach to Data Valuation for Federated Learning](https://mlsanity.com/paper/2009.06192/) (2020)
- [Foundation Models and Fair Use](https://mlsanity.com/paper/2303.15715/) (2023)
- [Sampling Permutations for Shapley Value Estimation](https://mlsanity.com/paper/2104.12199/) (2021)
- [What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions](https://mlsanity.com/paper/2405.13954/) (2024)
- [A Distributional Framework for Data Valuation](https://mlsanity.com/paper/2002.12334/) (2020)
- [LAVA: Data Valuation without Pre-Specified Learning Algorithms](https://mlsanity.com/paper/2305.00054/) (2023)
- [Differentially Private Optimization on Large Model at Small Cost](https://mlsanity.com/paper/2210.00038/) (2022)
- [OpenDataVal: a Unified Benchmark for Data Valuation](https://mlsanity.com/paper/2306.10577/) (2023)
- [Measuring the Effect of Training Data on Deep Learning Predictions via Randomized Experiments](https://mlsanity.com/paper/2206.10013/) (2022)

## Built on by (most cited first, 30 of 81)

- [Data Selection via Optimal Control for Language Models](https://mlsanity.com/paper/2410.07064/) (2024)
- [Capturing the Temporal Dependence of Training Data Influence](https://mlsanity.com/paper/2412.09538/) (2024)
- [OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration](https://mlsanity.com/paper/2602.05400/) (2026)
- [Regression-adjusted Monte Carlo Estimators for Shapley Values and Probabilistic Values](https://mlsanity.com/paper/2506.11849/) (2025)
- [DataRater: Meta-Learned Dataset Curation](https://mlsanity.com/paper/2505.17895/) (2025)
- [Shaping capabilities with token-level data filtering](https://mlsanity.com/paper/2601.21571/) (2026)
- [A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning](https://mlsanity.com/paper/2505.19281/) (2025)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Fairshare Data Pricing via Data Valuation for Large Language Models](https://mlsanity.com/paper/2502.00198/) (2025)
- [Fast Data Attribution for Text-to-Image Models](https://mlsanity.com/paper/2511.10721/) (2025)
- [MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging](https://mlsanity.com/paper/2601.17858/) (2026)
- [Train Once, Answer All: Many Pretraining Experiments for the Cost of One](https://mlsanity.com/paper/2509.23383/) (2025)
- [CAMs as Shapley Value-based Explainers](https://mlsanity.com/paper/2501.06261/) (2025)
- [DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models](https://mlsanity.com/paper/2507.09424/) (2025)
- [Motion Attribution for Video Generation](https://mlsanity.com/paper/2601.08828/) (2026)
- [Targeted synthetic data generation for tabular data via hardness characterization](https://mlsanity.com/paper/2410.00759/) (2024)
- [Scalable Valuation of Human Feedback through Provably Robust Model Alignment](https://mlsanity.com/paper/2505.17859/) (2025)
- [Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining](https://mlsanity.com/paper/2505.24261/) (2025)
- [Trade in Minutes! Rationality-Driven Agentic System for Quantitative Financial Trading](https://mlsanity.com/paper/2510.04787/) (2025)
- [How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines](https://mlsanity.com/paper/2605.18814/) (2026)
- [Concept-TRAK: Understanding how diffusion models learn concepts through concept-level attribution](https://mlsanity.com/paper/2507.06547/) (2025)
- [Z0-Inf: Zeroth Order Approximation for Data Influence](https://mlsanity.com/paper/2510.11832/) (2025)
- [MaxShapley: Towards Incentive-compatible Generative Search with Fair Context Attribution](https://mlsanity.com/paper/2512.05958/) (2025)
- [A Sustainable AI Economy Needs Data Deals That Work for Generators](https://mlsanity.com/paper/2601.09966/) (2026)
- [OATS: Online Data Augmentation for Time Series Foundation Models](https://mlsanity.com/paper/2601.19040/) (2026)
- [In-Run Data Shapley for Adam Optimizer](https://mlsanity.com/paper/2602.00329/) (2026)
- [ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws](https://mlsanity.com/paper/2603.17945/) (2026)
- [Do Data Valuations Make Good Data Prices?](https://mlsanity.com/paper/2504.05563/) (2025)
- [DICE: Data Influence Cascade in Decentralized Learning](https://mlsanity.com/paper/2507.06931/) (2025)
- [Newfluence: Boosting Model interpretability and Understanding in High Dimensions](https://mlsanity.com/paper/2507.11895/) (2025)

## Abstract (arXiv)

Data Shapley provides a principled framework for attributing data's contribution within machine learning contexts. However, existing approaches require re-training models on different data subsets, which is computationally intensive, foreclosing their application to large-scale models. Furthermore, they produce the same attribution score for any models produced by running the learning algorithm, meaning they cannot perform targeted attribution towards a specific model obtained from a single run of the algorithm. This paper introduces In-Run Data Shapley, which addresses these limitations by offering scalable data attribution for a target model of interest. In its most efficient implementation, our technique incurs negligible additional runtime compared to standard model training. This dramatic efficiency improvement makes it possible to perform data attribution for the foundation model pretraining stage for the first time. We present several case studies that offer fresh insights into pretraining data's contribution and discuss their implications for copyright in generative AI and pretraining data curation.