# Towards a statistical theory of data selection under weak supervision arXiv: https://arxiv.org/abs/2309.14563 · HTML: https://mlsanity.com/paper/2309.14563/ ## Key facts - Cited by 27 papers in the map (data through 14 Oct 2026). - Builds on 16 of its 32 references that are in the map. - First posted on arXiv 25 Sep 2023. ## Builds on (most cited first, 16 of 16) - [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020) - [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017) - [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021) - [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022) - [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019) - [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017) - [Accelerating Deep Learning by Focusing on the Biggest Losers](https://mlsanity.com/paper/1910.00762/) (2019) - [A Statistical Perspective on Algorithmic Leveraging](https://mlsanity.com/paper/1306.5362/) (2013) - [Less Is Better: Unweighted Data Subsampling via Influence Function](https://mlsanity.com/paper/1912.01321/) (2019) - [Optimal Subsampling for Large Sample Logistic Regression](https://mlsanity.com/paper/1702.01166/) (2017) - [A Statistical Perspective on Randomized Sketching for Ordinary Least-Squares](https://mlsanity.com/paper/1406.5986/) (2014) - [Are All Training Examples Created Equal? An Empirical Study](https://mlsanity.com/paper/1811.12569/) (2018) - [Optimal Sub-sampling with Influence Functions](https://mlsanity.com/paper/1709.01716/) (2017) - [Optimal Subsampling Algorithms for Big Data Regressions](https://mlsanity.com/paper/1806.06761/) (2018) - [Asymptotic Analysis of Sampling Estimators for Randomized Numerical Linear Algebra Algorithms](https://mlsanity.com/paper/2002.10526/) (2020) - [Large deviations for the perceptron model and consequences for active learning](https://mlsanity.com/paper/1912.03927/) (2019) ## Built on by (most cited first, 27 of 27) - [Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification](https://mlsanity.com/paper/2406.07515/) (2024) - [Most Influential Subset Selection: Challenges, Promises, and Beyond](https://mlsanity.com/paper/2409.18153/) (2024) - [SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching](https://mlsanity.com/paper/2406.18561/) (2024) - [High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws](https://mlsanity.com/paper/2410.18837/) (2024) - [A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspective](https://mlsanity.com/paper/2509.16499/) (2025) - [Improving the Scaling Laws of Synthetic Data with Deliberate Practice](https://mlsanity.com/paper/2502.15588/) (2025) - [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024) - [DRoP: Distributionally Robust Data Pruning](https://mlsanity.com/paper/2404.05579/) (2024) - [High-dimensional Analysis of Synthetic Data Selection](https://mlsanity.com/paper/2510.08123/) (2025) - [Sketchy Moment Matching: Toward Fast and Provable Data Selection for Finetuning](https://mlsanity.com/paper/2407.06120/) (2024) - [High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization](https://mlsanity.com/paper/2406.03171/) (2024) - [The Impact of Coreset Selection on Spurious Correlations and Group Robustness](https://mlsanity.com/paper/2507.11690/) (2025) - [Weak-to-Strong Generalization with Failure Trajectories: A Tree-based Approach to Elicit Optimal Policy in Strong Models](https://mlsanity.com/paper/2507.18858/) (2025) - [Train on Validation (ToV): Fast data selection with applications to fine-tuning](https://mlsanity.com/paper/2510.00386/) (2025) - [Balancing Label Quantity and Quality for Scalable Elicitation](https://mlsanity.com/paper/2410.13215/) (2024) - [RL-Guided Data Selection for Language Model Finetuning](https://mlsanity.com/paper/2509.25850/) (2025) - [MixMax: Distributional Robustness in Function Space via Optimal Data Mixtures](https://mlsanity.com/paper/2406.01477/) (2024) - [Optimal Regularization for Performative Learning](https://mlsanity.com/paper/2510.12249/) (2025) - [Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing](https://mlsanity.com/paper/2605.20105/) (2026) - [Backbone Augmented Training for Adaptations](https://mlsanity.com/paper/2506.04288/) (2025) - [Data-Efficient Training by Evolved Sampling](https://mlsanity.com/paper/2509.23461/) (2025) - [SNAP: Low-Latency Test-Time Adaptation with Sparse Updates](https://mlsanity.com/paper/2511.15276/) (2025) - [Understanding the Gain from Data Filtering in Multimodal Contrastive Learning](https://mlsanity.com/paper/2512.14230/) (2025) - [Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation](https://mlsanity.com/paper/2606.25927/) (2026) - [Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics](https://mlsanity.com/paper/2607.09250/) (2026) - [High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning](https://mlsanity.com/paper/2610.02578/) (2026) - [Measure Less, Know More: Self-Supervised Test-Time Feature Acquisition](https://mlsanity.com/paper/2610.03454/) (2026) ## Abstract (arXiv) Given a sample of size N, it is often useful to select a subsample of smaller size n