# Optimizing Data Collection for Machine Learning

arXiv: https://arxiv.org/abs/2210.01234 · HTML: https://mlsanity.com/paper/2210.01234/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 66 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Variational Adversarial Active Learning](https://mlsanity.com/paper/1904.00370/) (2019)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++](https://mlsanity.com/paper/1803.09693/) (2018)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [f-Domain-Adversarial Learning: Theory and Algorithms](https://mlsanity.com/paper/2106.11344/) (2021)
- [NeurIPS 2020 Competition: Predicting Generalization in Deep Learning](https://mlsanity.com/paper/2012.07976/) (2020)
- [Low Budget Active Learning via Wasserstein Distance: An Integer Programming Approach](https://mlsanity.com/paper/2106.02968/) (2021)
- [Not All Labels Are Equal: Rationalizing The Labeling Costs for Training Object Detection](https://mlsanity.com/paper/2106.11921/) (2021)
- [Active Learning under Label Shift](https://mlsanity.com/paper/2007.08479/) (2020)
- [How Much More Data Do I Need? Estimating Requirements for Downstream Tasks](https://mlsanity.com/paper/2207.01725/) (2022)
- [Learning Curves for Analysis of Deep Networks](https://mlsanity.com/paper/2010.11029/) (2020)
- [Self-Supervised Real-to-Sim Scene Generation](https://mlsanity.com/paper/2011.14488/) (2020)
- [Domain Adversarial Training: A Game Perspective](https://mlsanity.com/paper/2202.05352/) (2022)
- [A Theoretical-Empirical Approach to Estimating Sample Complexity of DNNs](https://mlsanity.com/paper/2105.01867/) (2021)
- [Optimal Data Collection for Randomized Control Trials](https://mlsanity.com/paper/1603.03675/) (2016)

## Built on by (most cited first, 15 of 15)

- [How to Select Which Active Learning Strategy is Best Suited for Your Specific Problem and Budget](https://mlsanity.com/paper/2306.03543/) (2023)
- [Scaling Laws for Hyperparameter Optimization](https://mlsanity.com/paper/2302.00441/) (2023)
- [Data Scaling Laws for End-to-End Autonomous Driving](https://mlsanity.com/paper/2504.04338/) (2025)
- [Bridging the Sim2Real gap with CARE: Supervised Detection Adaptation with Conditional Alignment and Reweighting](https://mlsanity.com/paper/2302.04832/) (2023)
- [Pricing and Competition for Generative AI](https://mlsanity.com/paper/2411.02661/) (2024)
- [Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns](https://mlsanity.com/paper/2303.11678/) (2023)
- [A Meta-Learning Approach to Predicting Performance and Data Requirements](https://mlsanity.com/paper/2303.01598/) (2023)
- [A Probabilistic Method to Predict Classifier Accuracy on Larger Datasets given Small Pilot Data](https://mlsanity.com/paper/2311.18025/) (2023)
- [AdaDeDup: Adaptive Hybrid Data Pruning for Efficient Large-Scale Object Detection Training](https://mlsanity.com/paper/2507.00049/) (2025)
- [Scalable Modular Synthetic Data Generation for Advancing Aerial Autonomy](https://mlsanity.com/paper/2211.05335/) (2022)
- [SelectFormer: Private and Practical Data Selection for Transformers](https://mlsanity.com/paper/2310.02373/) (2023)
- [LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought](https://mlsanity.com/paper/2505.15657/) (2025)
- [Characterization of Transfer Using Multi-task Learning Curves](https://mlsanity.com/paper/2512.24866/) (2025)
- [Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems](https://mlsanity.com/paper/2604.08366/) (2026)
- [Strategies and impact of learning curve estimation for CNN-based image classification](https://mlsanity.com/paper/2310.08470/) (2023)

## Abstract (arXiv)

Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what kind of data to collect. Over-collecting data incurs unnecessary present costs, while under-collecting may incur future costs and delay workflows. We propose a new paradigm for modeling the data collection workflow as a formal optimal data collection problem that allows designers to specify performance targets, collection costs, a time horizon, and penalties for failing to meet the targets. Additionally, this formulation generalizes to tasks requiring multiple data sources, such as labeled and unlabeled data used in semi-supervised learning. To solve our problem, we develop Learn-Optimize-Collect (LOC), which minimizes expected future collection costs. Finally, we numerically compare our framework to the conventional baseline of estimating data requirements by extrapolating from neural scaling laws. We significantly reduce the risks of failing to meet desired performance targets on several classification, segmentation, and detection tasks, while maintaining low total collection costs.