mlsanityData to 14 Oct 2026

Can pre-trained models assist in dataset distillation?

arXiv 2310.03295 ↗First posted 5 Oct 2023cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Dataset Distillation (DD) is a prominent technique that encapsulates knowledge from a large-scale original dataset into a small synthetic dataset for efficient training. Meanwhile, Pre-trained Models (PTMs) function as knowledge repositories, containing extensive information from the original dataset. This naturally raises a question: Can PTMs effectively transfer knowledge to synthetic datasets, guiding DD accurately? To this end, we conduct preliminary experiments, confirming the contribution of PTMs to DD. Afterwards, we systematically study different options in PTMs, including initialization parameters, model architecture, training epoch and domain knowledge, revealing that: 1) Increasing model diversity enhances the performance of synthetic datasets; 2) Sub-optimal models can also assist in DD and outperform well-trained ones in certain cases; 3) Domain-specific PTMs are not mandatory for DD, but a reasonable domain match is crucial. Finally, by selecting optimal options, we significantly improve the cross-architecture generalization over baseline DD methods. We hope our work will facilitate researchers to develop better DD techniques. Our code is available at https://github.com/yaolu-zjut/DDInterpreter.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 46 of its 58 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

46 of 58 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  7. 2020Supervised Contrastive Learningcited by 2,829
  8. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  9. 2020Knowledge Distillation: A Surveycited by 1,603
  10. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  11. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  12. 2020A Survey on Contrastive Self-supervised Learningcited by 554
34 more
  1. 2020Dataset Condensation with Gradient Matchingcited by 516
  2. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  3. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  4. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 377
  5. 2021Dataset Condensation with Distribution Matchingcited by 365
  6. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  7. 2013OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networkscited by 224
  8. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 222
  9. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 218
  10. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 188
  11. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  12. 2020Domain Adaptive Ensemble Learningcited by 156
  13. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 144
  14. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  15. 2023Generalizing Dataset Distillation via Deep Generative Priorcited by 126
  16. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 120
  17. 2023A Comprehensive Survey of Dataset Distillationcited by 115
  18. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 104
  19. 2022Condensing Graphs via One-Step Gradient Matchingcited by 100
  20. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 94
  21. 2020Understanding and Improving Knowledge Distillationcited by 83
  22. 2018TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentationcited by 76
  23. 2022DC-BENCH: Dataset Condensation Benchmarkcited by 67
  24. 2022FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learningcited by 66
  25. 2022Expanding Small-Scale Datasets with Guided Imaginationcited by 62
  26. 2021Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuningcited by 50
  27. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  28. 2023Dataset Quantizationcited by 48
  29. 2023DiM: Distilling Dataset into Generative Modelcited by 46
  30. 2022FedSynth: Gradient Compression via Synthetic Data in Federated Learningcited by 31
  31. 2022Meta Knowledge Condensation for Federated Learningcited by 26
  32. 2022Self-Supervised Pretraining for 2D Medical Image Segmentationcited by 24
  33. 2022Infinite Recommendation Networks: A Data-Centric Approachcited by 20
  34. 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 5; 2026: 1.

12 papers.

  1. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 109
  2. 2024Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matchingcited by 28
  3. 2025The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutionscited by 26
  4. 2025TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendationcited by 13
  5. 2024Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matchingcited by 8
  6. 2024GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruningcited by 8
  7. 2024Exploring the Impact of Dataset Bias on Dataset Distillationcited by 4
  8. 2025Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reportscited by 1
  9. 2026Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Spacecited by 1
  10. 2024MDM: Advancing Multi-Domain Distribution Matching for Automatic Modulation Recognition Dataset Synthesiscited by 0
  11. 2025Multi-Modal Machine Learning Framework for Predicting Early Recurrence of Brain Tumors Using MRI and Clinical Biomarkerscited by 0
  12. 2025A Multimodal Deep Learning Framework for Early Diagnosis of Liver Cancer via Optimized BiLSTM-AM-VMD Architecturecited by 0