mlsanityData to 7 Oct 2026

Rethinking Data Distillation: Do Not Overlook Calibration

arXiv 2307.12463 ↗First posted 24 Jul 2023cs.CV cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Neural networks trained on distilled data often produce over-confident output and require correction by calibration methods. Existing calibration methods such as temperature scaling and mixup work well for networks trained on original large-scale data. However, we find that these methods fail to calibrate networks trained on data distilled from large source datasets. In this paper, we show that distilled data lead to networks that are not calibratable due to (i) a more concentrated distribution of the maximum logits and (ii) the loss of information that is semantically meaningful but unrelated to classification tasks. To address this problem, we propose Masked Temperature Scaling (MTS) and Masked Distillation Training (MDT) which mitigate the limitations of distilled data and achieve better calibration results while maintaining the efficiency of dataset distillation.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 35 of its 58 references that are in the map. First posted on arXiv 24 Jul 2023.

Builds on

35 of 58 references.

References · 35

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  6. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  7. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  8. 2019When Does Label Smoothing Help?cited by 985
  9. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  10. 2017Focal Loss for Dense Object Detectioncited by 749
  11. 2020Dataset Condensation with Gradient Matchingcited by 510
  12. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
23 more
  1. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  2. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  3. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  4. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 251
  5. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 231
  6. 2020Dataset Meta-Learning from Kernel Ridge-Regressioncited by 230
  7. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  8. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 187
  9. 2022Dataset Distillation using Neural Feature Regressioncited by 177
  10. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  11. 2020Improving model calibration with accuracy versus uncertainty optimizationcited by 123
  12. 2022Efficient Dataset Distillation Using Random Feature Approximationcited by 111
  13. 2019Soft-Label Dataset Distillation and Text Dataset Distillationcited by 106
  14. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  15. 2022Remember the Past: Distilling Datasets into Addressable Memories for Neural Networkscited by 104
  16. 2021Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methodscited by 83
  17. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  18. 2018Self-supervised Knowledge Distillation Using Singular Value Decompositioncited by 47
  19. 2015Dropout as data augmentationcited by 27
  20. 2022IDEAL: Query-Efficient Data-Free Learning from Black-box Modelscited by 16
  21. 2022On Calibrated Model Uncertainty in Deep Learningcited by 7
  22. 2023Towards Reliable Rare Category Analysis on Graphs via Individual Calibrationcited by 6
  23. 2023Calibrating the Rigged Lottery: Making All Tickets Reliablecited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4; 2025: 7.

13 papers.

Sort:

By area:

  1. 2024Generative Dataset Distillation Based on Diffusion Modelcited by 13
  2. 2023Dataset Distillation in Latent Spacecited by 7
  3. 2023Data Optimization in Deep Learning: A Surveycited by 6
  4. 2025Optimizing Distributional Geometry Alignment with Optimal Transport for Generative Dataset Distillationcited by 4
  5. 2024Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful Learningcited by 3
  6. 2025Distilling Calibration via Conformalized Credal Inferencecited by 2
  7. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  8. 2024Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real Worldcited by 1
  9. 2024Data-Efficient Generation for Dataset Distillationcited by 0
  10. 2025Dataset Distillation with Probabilistic Latent Featurescited by 0
  11. 2025Video Dataset Condensation with Diffusion Modelscited by 0
  12. 2025Leveraging Multi-Modal Information to Enhance Dataset Distillationcited by 0
1 more
  1. 2025MAJORScore: A Novel Metric for Evaluating Multimodal Relevance via Joint Representationcited by 0