mlsanityData to 14 Oct 2026

Towards a statistical theory of data selection under weak supervision

arXiv 2309.14563 ↗First posted 25 Sep 2023stat.ML cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

Given a sample of size N, it is often useful to select a subsample of smaller size n<N to be used for statistical estimation or learning. Such a data selection step is useful to reduce the requirements of data labeling and the computational complexity of learning. We assume to be given N unlabeled samples boldsymbol xii≤ N, and to be given access to a `surrogate model' that can predict labels yi better than random guessing. Our goal is to select a subset of the samples, to be denoted by boldsymbol xii∈ G, of size |G|=n<N. We then acquire labels for this set and we use them to train a model via regularized empirical risk minimization. By using a mixture of numerical experiments on real and synthetic data, and mathematical derivations under low- and high- dimensional asymptotics, we show that: (i)~Data selection can be very effective, in particular beating training on the full sample in some cases; (ii)~Certain popular choices in data selection methods (e.g. unbiased reweighted subsampling, or influence function-based subsampling) can be substantially suboptimal.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

16 of 32 references.

References · 16

  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  2. 2017Deep Bayesian Active Learning with Image Datacited by 819
  3. 2021KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3Dcited by 575
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  6. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  7. 2019Accelerating Deep Learning by Focusing on the Biggest Loserscited by 94
  8. 2013A Statistical Perspective on Algorithmic Leveragingcited by 39
  9. 2019Less Is Better: Unweighted Data Subsampling via Influence Functioncited by 30
  10. 2017Optimal Subsampling for Large Sample Logistic Regressioncited by 29
  11. 2014A Statistical Perspective on Randomized Sketching for Ordinary Least-Squarescited by 21
  12. 2018Are All Training Examples Created Equal? An Empirical Studycited by 16
4 more
  1. 2017Optimal Sub-sampling with Influence Functionscited by 11
  2. 2018Optimal Subsampling Algorithms for Big Data Regressionscited by 11
  3. 2020Asymptotic Analysis of Sampling Estimators for Randomized Numerical Linear Algebra Algorithmscited by 10
  4. 2019Large deviations for the perceptron model and consequences for active learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 12; 2026: 5.

27 papers.

Sort:

By area:

  1. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
  2. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 32
  3. 2024SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matchingcited by 22
  4. 2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 22
  5. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  6. 2025Improving the Scaling Laws of Synthetic Data with Deliberate Practicecited by 17
  7. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  8. 2024DRoP: Distributionally Robust Data Pruningcited by 7
  9. 2025High-dimensional Analysis of Synthetic Data Selectioncited by 7
  10. 2024Sketchy Moment Matching: Toward Fast and Provable Data Selection for Finetuningcited by 5
  11. 2024High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularizationcited by 4
  12. 2025The Impact of Coreset Selection on Spurious Correlations and Group Robustnesscited by 4
15 more
  1. 2025Weak-to-Strong Generalization with Failure Trajectories: A Tree-based Approach to Elicit Optimal Policy in Strong Modelscited by 3
  2. 2025Train on Validation (ToV): Fast data selection with applications to fine-tuningcited by 3
  3. 2024Balancing Label Quantity and Quality for Scalable Elicitationcited by 2
  4. 2025RL-Guided Data Selection for Language Model Finetuningcited by 2
  5. 2024MixMax: Distributional Robustness in Function Space via Optimal Data Mixturescited by 1
  6. 2025Optimal Regularization for Performative Learningcited by 1
  7. 2026Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probingcited by 1
  8. 2025Backbone Augmented Training for Adaptationscited by 0
  9. 2025Data-Efficient Training by Evolved Samplingcited by 0
  10. 2025SNAP: Low-Latency Test-Time Adaptation with Sparse Updatescited by 0
  11. 2025Understanding the Gain from Data Filtering in Multimodal Contrastive Learningcited by 0
  12. 2026Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimationcited by 0
  13. 2026Influence Diagnostics in High-dimensional M-estimation: Precise Asymptoticscited by 0
  14. 2026High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learningcited by 0
  15. 2026Measure Less, Know More: Self-Supervised Test-Time Feature Acquisitioncited by 0