mlsanityData to 14 Oct 2026

Unifying Dataset Pruning and Distillation for Efficient Large-scale Compression

arXiv 2502.06434 ↗First posted 10 Feb 2025cs.CV cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Dataset pruning (DP) and dataset distillation (DD) fundamentally differ in their outputs: DP selects original image subsets, while DD generates synthetic images. Recently, DD's increasing reliance on original images suggests a convergence of the two directions. To investigate this convergence trend, we propose a unified dataset compression (DC) benchmark. This benchmark reveals an interesting trade-off for soft-label-DD: while soft labels provide valuable information, they can make the distillation process less essential, as distilled images may not always outperform random subsets. In addition, the benchmark reveals that in current stages, dataset pruning outperforms dataset distillation at small dataset sizes. Given these observations, we explore hard-label-DC as a complementary approach that emphasizes image quality while offering substantial storage efficiency. Our PCA (Prune, Combine, and Augment) is the first framework that does not rely on soft labels but instead focuses on image quality. (1) "P'' means selecting easy samples based on dataset pruning metrics, (2) "C'' indicates combining these samples effectively, and (3) "A'' is to apply constrained image augmentation during training. Our code is available at https://github.com/ArmandXiao/Unifying-Dataset-Pruning-and-Distillation

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 59 of its 73 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

59 of 73 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  7. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  8. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  9. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  10. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  11. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  12. 2020Dataset Condensation with Gradient Matchingcited by 516
38 more
  1. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  2. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  3. 2021Dataset Condensation with Distribution Matchingcited by 365
  4. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  5. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  6. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  7. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  8. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
  9. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  10. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  11. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 188
  12. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 181
  13. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  14. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  15. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 138
  16. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 133
  17. 2022Coverage-centric Coreset Selection for High Pruning Ratescited by 117
  18. 2022Dataset Distillation via Factorizationcited by 115
  19. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  20. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 109
  21. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  22. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 94
  23. 2022Synthesizing Informative Training Samples with GANcited by 77
  24. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 77
  25. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  26. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  27. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  28. 2023DiM: Distilling Dataset into Generative Modelcited by 46
  29. 2022Delving into Effective Gradient Matching for Dataset Condensationcited by 40
  30. 2024Elucidating the Design Space of Dataset Condensationcited by 40
  31. 2023Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruningcited by 38
  32. 2022Dataset Condensation with Latent Space Knowledge Factorization and Sharingcited by 34
  33. 2024Effective pruning of web-scale datasets based on complexity of concept clusterscited by 34
  34. 2023Loss-Curvature Matching for Dataset Selection and Condensationcited by 33
  35. 2024Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustmentcited by 33
  36. 2024A Label is Worth a Thousand Images in Dataset Distillationcited by 31
  37. 2023Dataset Distillation via Curriculum Data Synthesis in Large Data Eracited by 26
  38. 2023You Only Condense Once: Two Rules for Pruning Condensed Datasetscited by 20

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025A Coreset Selection of Coreset Selection Literature: Introduction and Recent Advancescited by 28
  2. 2025Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluationcited by 1
  3. 2025HyperCore: Coreset Selection under Noise via Hypersphere Modelscited by 0
  4. 2025SubZeroCore: A Submodular Approach with Zero Training for Coreset Selectioncited by 0
  5. 2026Accelerating Large-Scale Dataset Distillation via Exploration-Exploitation Optimizationcited by 0
  6. 2026Soft Label Pruning and Quantization for Large-Scale Dataset Distillationcited by 0