mlsanityData to 7 Oct 2026

Selection via Proxy: Efficient Data Selection for Deep Learning

arXiv 1906.11829 ↗First posted 26 Jun 2019cs.LG stat.MLcited by 337

Not on the map. Why?

Abstract (arXiv)

Data selection methods, such as active learning and core-set selection, are useful tools for machine learning on large datasets. However, they can be prohibitively expensive to apply in deep learning because they depend on feature representations that need to be learned. In this work, we show that we can greatly improve the computational efficiency by using a small proxy model to perform data selection (e.g., selecting data points to label for active learning). By removing hidden layers from the target model, using smaller architectures, and training for fewer epochs, we create proxies that are an order of magnitude faster to train. Although these small proxy models have higher error rates, we find that they empirically provide useful signals for data selection. We evaluate this "selection via proxy" (SVP) approach on several data selection tasks across five datasets: CIFAR10, CIFAR100, ImageNet, Amazon Review Polarity, and Amazon Review Full. For active learning, applying SVP can give an order of magnitude improvement in data selection runtime (i.e., the time it takes to repeatedly train and select points) without significantly increasing the final error (often within 0.1%). For core-set selection on CIFAR10, proxies that are over 10x faster to train than their larger, more accurate targets can remove up to 50% of the data without harming the final accuracy of the target, leading to a 1.6x end-to-end training time improvement.

Key facts. Cited by 337 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 30; 2023: 79; 2024: 82; 2025: 74; 2026: 52.

337 papers.

Sort:

By area:

  1. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  2. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  3. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  4. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  5. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  6. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  7. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  8. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  9. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  10. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 146
  11. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  12. 2024Rho-1: Not All Tokens Are What You Needcited by 132
38 more
  1. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  2. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 115
  3. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  4. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 107
  5. 2023Finding Support Examples for In-Context Learningcited by 104
  6. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  7. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 93
  8. 2024How to Train Data-Efficient LLMscited by 91
  9. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  10. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  11. 2023Data Distillation: A Surveycited by 86
  12. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  13. 2021On Statistical Bias In Active Learning: How and When To Fix Itcited by 72
  14. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  15. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  16. 2023Data Pruning via Moving-one-Sample-outcited by 67
  17. 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
  18. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  19. 2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
  20. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  21. 2023M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancycited by 49
  22. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 47
  23. 2023Dataset Quantizationcited by 47
  24. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  25. 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
  26. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearningcited by 44
  27. 2021Accelerating Deep Learning with Dynamic Data Pruningcited by 43
  28. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  29. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 42
  30. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  31. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  32. 2023Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruningcited by 38
  33. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  34. 2021Self-Supervised Pre-Training for Transformer-Based Person Re-Identificationcited by 36
  35. 2022ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Datacited by 34
  36. 2024Effective pruning of web-scale datasets based on complexity of concept clusterscited by 34
  37. 2023Loss-Curvature Matching for Dataset Selection and Condensationcited by 32
  38. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32

50 most cited of 337 All 337 (JSON)