# Scaling Laws for the Value of Individual Data Points in Machine Learning

arXiv: https://arxiv.org/abs/2405.20456 · HTML: https://mlsanity.com/paper/2405.20456/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning](https://mlsanity.com/paper/2110.14049/) (2021)
- [DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models](https://mlsanity.com/paper/2310.00902/) (2023)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [A Distributional Framework for Data Valuation](https://mlsanity.com/paper/2002.12334/) (2020)
- [LAVA: Data Valuation without Pre-Specified Learning Algorithms](https://mlsanity.com/paper/2305.00054/) (2023)
- [OpenDataVal: a Unified Benchmark for Data Valuation](https://mlsanity.com/paper/2306.10577/) (2023)
- [Data-OOB: Out-of-bag Estimate as a Simple and Efficient Data Value](https://mlsanity.com/paper/2304.07718/) (2023)
- [Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data](https://mlsanity.com/paper/2103.03399/) (2021)
- [Variance reduced Shapley value estimation for trustworthy data valuation](https://mlsanity.com/paper/2210.16835/) (2022)
- [Evaluating Self-Supervised Learning via Risk Decomposition](https://mlsanity.com/paper/2302.03068/) (2023)
- [Deterministic Inequalities for Smooth M-estimators](https://mlsanity.com/paper/1809.05172/) (2018)

## Built on by (most cited first, 10 of 10)

- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [(Mis)Fitting: A Survey of Scaling Laws](https://mlsanity.com/paper/2502.18969/) (2025)
- [Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder](https://mlsanity.com/paper/2502.14050/) (2025)
- [How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines](https://mlsanity.com/paper/2502.12051/) (2025)
- [Safety vs. Performance: How Multi-Objective Learning Reduces Barriers to Market Entry](https://mlsanity.com/paper/2409.03734/) (2024)
- [AI Evaluation Should Require Standardized Item-Level Data Releases](https://mlsanity.com/paper/2604.03244/) (2026)
- [Unsupervised Data Validation Methods for Efficient Model Training](https://mlsanity.com/paper/2410.07880/) (2024)
- [DROP: Poison Dilution via Knowledge Distillation for Federated Learning](https://mlsanity.com/paper/2502.07011/) (2025)
- [Physics-Informed Uncertainty Enables Reliable AI-driven Design](https://mlsanity.com/paper/2601.18638/) (2026)
- [On the Fragility of Data Attribution When Learning Is Distributed](https://mlsanity.com/paper/2605.15520/) (2026)

## Abstract (arXiv)

Recent works have shown that machine learning models improve at a predictable rate with the total amount of training data, leading to scaling laws that describe the relationship between error and dataset size. These scaling laws can help design a model's training dataset, but they typically take an aggregate view of the data by only considering the dataset's size. We introduce a new perspective by investigating scaling behavior for the value of individual data points: we find that a data point's contribution to model's performance shrinks predictably with the size of the dataset in a log-linear manner. Interestingly, there is significant variability in the scaling exponent among different data points, indicating that certain points are more valuable in small datasets while others are relatively more useful as a part of large datasets. We provide learning theory to support our scaling law, and we observe empirically that it holds across diverse model classes. We further propose a maximum likelihood estimator and an amortized estimator to efficiently learn the individualized scaling behaviors from a small number of noisy observations per data point. Using our estimators, we provide insights into factors that influence the scaling behavior of different data points. Finally, we demonstrate applications of the individualized scaling laws to data valuation and data subset selection. Overall, our work represents a first step towards understanding and utilizing scaling properties for the value of individual data points.