mlsanityData to 7 Oct 2026

GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training

arXiv 2103.00123 ↗First posted 27 Feb 2021cs.LGcited by 269

Not on the map. Why?

Abstract (arXiv)

The great success of modern machine learning models on large datasets is contingent on extensive computational resources with high financial and environmental costs. One way to address this is by extracting subsets that generalize on par with the full data. In this work, we propose a general framework, GRAD-MATCH, which finds subsets that closely match the gradient of the training or validation set. We find such subsets effectively using an orthogonal matching pursuit algorithm. We show rigorous theoretical and convergence guarantees of the proposed algorithm and, through our extensive experiments on real-world datasets, show the effectiveness of our proposed framework. We show that GRAD-MATCH significantly and consistently outperforms several recent data-selection algorithms and achieves the best accuracy-efficiency trade-off. GRAD-MATCH is available as a part of the CORDS toolkit: urlhttps://github.com/decile-team/cords.

Key facts. Cited by 269 papers in the map (data through 7 Oct 2026). Builds on 16 of its 41 references that are in the map. First posted on arXiv 27 Feb 2021.

Builds on

16 of 41 references.

References · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  5. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  6. 2019Green AIcited by 620
  7. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  8. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  9. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  10. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
  11. 2018Bayesian Coreset Construction via Greedy Iterative Geodesic Ascentcited by 68
  12. 2019Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Visioncited by 64
4 more
  1. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  2. 2016Restricted Strong Convexity Implies Weak Submodularitycited by 51
  3. 2019E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savingscited by 36
  4. 2020Coresets for Robust Training of Neural Networks against Noisy Labelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 19; 2023: 50; 2024: 55; 2025: 75; 2026: 66.

269 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  3. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  4. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 146
  5. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 115
  6. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  7. 2023Finding Support Examples for In-Context Learningcited by 104
  8. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  9. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  10. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  11. 2023InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruningcited by 90
  12. 2023Data Distillation: A Surveycited by 86
38 more
  1. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
  2. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  3. 2022Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Trainingcited by 78
  4. 2023Data Pruning via Moving-one-Sample-outcited by 67
  5. 2023Dataset Quantizationcited by 47
  6. 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
  7. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  8. 2023Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruningcited by 38
  9. 2023Loss-Curvature Matching for Dataset Selection and Condensationcited by 32
  10. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  11. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  12. 2024TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Datacited by 28
  13. 2025A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advancescited by 28
  14. 2021TALISMAN: Targeted Active Learning for Object Detection with Rare Classes and Slices using Submodular Mutual Informationcited by 27
  15. 2023Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learningcited by 26
  16. 2022AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuningcited by 22
  17. 2023Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Leastcited by 22
  18. 2024SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matchingcited by 22
  19. 2022Mitigating Dataset Bias by Using Per-sample Gradientcited by 21
  20. 2023Regularizing Second-Order Influences for Continual Learningcited by 21
  21. 2022Infinite Recommendation Networks: A Data-Centric Approachcited by 20
  22. 2023You Only Condense Once: Two Rules for Pruning Condensed Datasetscited by 20
  23. 2023Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracycited by 20
  24. 2024A CLIP-Powered Framework for Robust and Generalizable Data Selectioncited by 20
  25. 2023Practical Differentially Private Hyperparameter Tuning with Subsamplingcited by 19
  26. 2024SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuningcited by 19
  27. 2025Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learningcited by 19
  28. 2024Importance-Aware Adaptive Dataset Distillationcited by 18
  29. 2024Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Modelscited by 17
  30. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 17
  31. 2022CVNets: High Performance Library for Computer Visioncited by 16
  32. 2025Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgetscited by 16
  33. 2023Iterative Forward Tuning Boosts In-Context Learning in Language Modelscited by 15
  34. 2023Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selectioncited by 14
  35. 2023Provable Data Subset Selection For Efficient Neural Network Trainingcited by 14
  36. 2024DELIFT: Data Efficient Language model Instruction Fine Tuningcited by 14
  37. 2025Data Pruning by Information Maximizationcited by 14
  38. 2023Less is More: Data Pruning for Faster Adversarial Trainingcited by 13

50 most cited of 269 All 269 (JSON)