# How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions

arXiv: https://arxiv.org/abs/2605.29448 · HTML: https://mlsanity.com/paper/2605.29448/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 103 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 20 of 20)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster](https://mlsanity.com/paper/2304.03208/) (2023)
- [Resolving Discrepancies in Compute-Optimal Scaling of Language Models](https://mlsanity.com/paper/2406.19146/) (2024)
- [One Model To Learn Them All](https://mlsanity.com/paper/1706.05137/) (2017)
- [Practical Coreset Constructions for Machine Learning](https://mlsanity.com/paper/1703.06476/) (2017)
- [Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning](https://mlsanity.com/paper/2505.20161/) (2025)
- [Reconciling Kaplan and Chinchilla Scaling Laws](https://mlsanity.com/paper/2406.12907/) (2024)
- [Submodularity In Machine Learning and Artificial Intelligence](https://mlsanity.com/paper/2202.00132/) (2022)
- [Depthwise Separable Convolutions for Neural Machine Translation](https://mlsanity.com/paper/1706.03059/) (2017)
- [AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs](https://mlsanity.com/paper/2407.20177/) (2024)
- [Private Data Valuation and Fair Payment in Data Marketplaces](https://mlsanity.com/paper/2210.08723/) (2022)
- [Deep Submodular Functions](https://mlsanity.com/paper/1701.08939/) (2017)
- [A Bitter Lesson for Data Filtering](https://mlsanity.com/paper/2605.19407/) (2026)
- [Non-submodular Function Maximization subject to a Matroid Constraint, with Applications](https://mlsanity.com/paper/1811.07863/) (2018)
- [PrivaDE: Privacy-preserving Data Evaluation for Blockchain-based Data Marketplaces](https://mlsanity.com/paper/2510.18109/) (2025)
- [Vendi Novelty Scores for Out-of-Distribution Detection](https://mlsanity.com/paper/2602.10062/) (2026)
- [Diversity You Can Actually Measure: A Fast, Model-Free Diversity Metric for Robotics Datasets](https://mlsanity.com/paper/2603.11634/) (2026)

## Abstract (arXiv)

Neural scaling laws appraise data through dataset size, while the Vendi Score uses quantum entropy to measure dataset value. We show both that common neural-scaling-law objectives and the Vendi Score are submodular. We further show that the Vendi Score is a special case of a broader class of submodular objectives that we call matrix spectral functions. This also includes determinantal (DPP) objectives, as well as many others. We also introduce weakly matrix monotone functions and show how they lead to weakly submodular matrix spectral functions, yielding a broad family of practical objectives for data appraisal. We develop secular-equation-based updates that avoid repeated eigendecompositions during greedy optimization, reducing marginal-gain evaluation for m-dimensional embeddings by an O(m) factor relative to oracle queries. This yields an average empirical speedup of about 35,000x, making direct optimization of the Vendi Score feasible on ImageNet-1K-scale datasets. Thus enabled, we compare how well several objectives predict the value of training subsets for held-out test performance under fixed-size, class-balanced, and fixed training-budget regimes, including the Vendi Score, DPPs, facility location, and three new matrix spectral variants. Across multiple datasets, facility location performs the best. Direct optimization also reveals that, while the Vendi Score is predictive over moderate score ranges, pushing the objective to higher values can make it a poor downstream performance proxy. We also find that uniformly at random fixed-size subsets, both unconstrained and class-balanced, are remarkably concentrated in both appraisal scores and held-out performance. Finally, we show that size, class balance, and training budget do not alone determine data value: even when controlling for these factors, performance ranges smoothly from good to bad.