mlsanityData to 7 Oct 2026

On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm

arXiv 2312.03526 ↗First posted 6 Dec 2023cs.CV cs.AI cs.LGcited by 107

Not on the map. Why?

Cited by 21 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contemporary machine learning requires training large neural networks on massive datasets and thus faces the challenges of high computational demands. Dataset distillation, as a recent emerging strategy, aims to compress real-world datasets for efficient training. However, this line of research currently struggle with large-scale and high-resolution datasets, hindering its practicality and feasibility. To this end, we re-examine the existing dataset distillation methods and identify three properties required for large-scale real-world applications, namely, realism, diversity, and efficiency. As a remedy, we propose RDED, a novel computationally-efficient yet effective data distillation paradigm, to enable both diversity and realism of the distilled data. Extensive empirical results over various neural architectures and datasets demonstrate the advancement of RDED: we can distill the full ImageNet-1K to a small dataset comprising 10 images per class within 7 minutes, achieving a notable 42% top-1 accuracy with ResNet-18 on a single RTX-4090 GPU (while the SOTA only achieves 21% but requires 6 hours).

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 45 of its 56 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

45 of 56 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  12. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
33 more
  1. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  2. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  3. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  4. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  5. 2020Dataset Condensation with Gradient Matchingcited by 510
  6. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  7. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  8. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  9. 2021Dataset Condensation with Distribution Matchingcited by 362
  10. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  12. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  13. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 180
  14. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  15. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  16. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  17. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 136
  18. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  19. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 125
  20. 2023Improved Distribution Matching for Dataset Condensationcited by 124
  21. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  22. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  23. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  24. 2016Understanding data augmentation for classification: when to warp?cited by 74
  25. 2023Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matchingcited by 69
  26. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  27. 2023Data Pruning via Moving-one-Sample-outcited by 67
  28. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  29. 2021A Fast Knowledge Distillation Framework for Visual Recognitioncited by 50
  30. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  31. 2021Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)cited by 35
  32. 2019Introduction to Coresets: Accurate Coresetscited by 17
  33. 2023Can pre-trained models assist in dataset distillation?cited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 54; 2026: 33.

107 papers.

Sort:

By area:

  1. 2025Dataset Distillation with Neural Characteristic Function: A Minmax Perspectivecited by 45
  2. 2024Elucidating the Design Space of Dataset Condensationcited by 40
  3. 2024Diversity-Driven Synthesis: Enhancing Dataset Distillation through Directed Weight Adjustmentcited by 32
  4. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  5. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 24
  6. 2025Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metriccited by 24
  7. 2024DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillationcited by 20
  8. 2024Multisize Dataset Condensationcited by 19
  9. 2024Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?cited by 19
  10. 2024Emphasizing Discriminative Features for Dataset Distillation in Complex Scenarioscited by 18
  11. 2025Dataset Distillation via Vision-Language Category Prototypecited by 17
  12. 2025OODD: Test-time Out-of-Distribution Detection with Dynamic Dictionarycited by 16
38 more
  1. 2025Taming Diffusion for Dataset Distillation with High Representativenesscited by 16
  2. 2025Hyperbolic Dataset Distillationcited by 15
  3. 2024Self-supervised Dataset Distillation: A Good Compression Is All You Needcited by 13
  4. 2024Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensatorcited by 13
  5. 2025Dataset Distillation via Committee Votingcited by 13
  6. 2025CaO2: Rectifying Inconsistencies in Diffusion-Based Dataset Distillationcited by 13
  7. 2024One-shot Federated Learning via Synthetic Distiller-Distillate Communicationcited by 12
  8. 2025FADRM: Fast and Accurate Data Residual Matching for Dataset Distillationcited by 11
  9. 2025Information-Guided Diffusion Sampling for Dataset Distillationcited by 10
  10. 2024GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Costcited by 9
  11. 2025Enhancing Dataset Distillation via Non-Critical Region Refinementcited by 9
  12. 2025DD-Ranking: Rethinking the Evaluation of Dataset Distillationcited by 8
  13. 2025Accelerating Diffusion Model Training under Minimal Budgets: A Condensation-Based Perspectivecited by 7
  14. 2026HIERAMP: Coarse-to-Fine Autoregressive Amplification for Generative Dataset Distillationcited by 7
  15. 2024SCAN: Bootstrapping Contrastive Pre-training for Data Efficiencycited by 6
  16. 2024A Large-Scale Study on Video Action Dataset Condensationcited by 6
  17. 2025Unifying Dataset Pruning and Distillation for Efficient Large-scale Compressioncited by 6
  18. 2025When Dynamic Data Selection Meets Data Augmentationcited by 6
  19. 2025Beyond Modality Collapse: Representations Blending for Multimodal Dataset Distillationcited by 6
  20. 2025MGD3: Mode-Guided Dataset Distillation using Diffusion Modelscited by 6
  21. 2025Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoringcited by 6
  22. 2025Efficient Multimodal Dataset Distillation via Generative Modelscited by 6
  23. 2024One Category One Prompt: Dataset Distillation using Diffusion Modelscited by 5
  24. 2024Label-Augmented Dataset Distillationcited by 5
  25. 2025Retrievals Can Be Detrimental: Unveiling the Backdoor Vulnerability of Retrieval-Augmented Diffusion Modelscited by 5
  26. 2025Distilling Dataset into Neural Fieldcited by 5
  27. 2025Dataset Condensation with Color Compensationcited by 5
  28. 2024Mitigating Bias in Dataset Distillationcited by 4
  29. 2024Prioritize Alignment in Dataset Distillationcited by 4
  30. 2024Heavy Labels Out! Dataset Distillation with Label Space Lighteningcited by 4
  31. 2025Beyond Random: Automatic Inner-loop Optimization in Dataset Distillationcited by 4
  32. 2025Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillationcited by 4
  33. 2024Towards Consistent and Efficient Dataset Distillation via Diffusion-Driven Selectioncited by 3
  34. 2025CONCORD: Concept-Informed Diffusion for Dataset Distillationcited by 3
  35. 2025OD3: Optimization-free Dataset Distillation for Object Detectioncited by 3
  36. 2025Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabelingcited by 3
  37. 2026EVLF: Early Vision-Language Fusion for Generative Dataset Distillationcited by 3
  38. 2026DMGD: Train-Free Dataset Distillation with Semantic-Distribution Matching in Diffusion Modelscited by 3

50 most cited of 107 All 107 (JSON)