mlsanityData to 14 Oct 2026

Revealing the Two Sides of Data Augmentation: An Asymmetric Distillation-based Win-Win Solution for Open-Set Recognition

arXiv 2404.19527 ↗First posted 28 Apr 2024cs.CVcited by 4

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

In this paper, we reveal the two sides of data augmentation: enhancements in closed-set recognition correlate with a significant decrease in open-set recognition. Through empirical investigation, we find that multi-sample-based augmentations would contribute to reducing feature discrimination, thereby diminishing the open-set criteria. Although knowledge distillation could impair the feature via imitation, the mixed feature with ambiguous semantics hinders the distillation. To this end, we propose an asymmetric distillation framework by feeding teacher model extra raw data to enlarge the benefit of teacher. Moreover, a joint mutual information loss and a selective relabel strategy are utilized to alleviate the influence of hard mixed samples. Our method successfully mitigates the decline in open-set and outperforms SOTAs by 2%~3% AUROC on the Tiny-ImageNet dataset and experiments on large-scale dataset ImageNet-21K demonstrate the generalization of our method.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 31 of its 48 references that are in the map. First posted on arXiv 28 Apr 2024.

Builds on

31 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Understanding the Role of Mixup in Knowledge Distillation: An Empirical Study8 Nov
  2. 2022Knowledge Distillation from A Stronger Teacher21 May

Other references · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  11. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  12. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
17 more
  1. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 690
  2. 2015Towards Open Set Deep Networkscited by 538
  3. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 353
  4. 2021Adversarial Reciprocal Points Learning for Open Set Recognitioncited by 153
  5. 2017Dataset Augmentation in Feature Spacecited by 124
  6. 2022Image Data Augmentation for Deep Learning: A Surveycited by 96
  7. 2022The Effects of Regularization and Data Augmentation are Class Dependentcited by 67
  8. 2017Smart Augmentation - Learning an Optimal Data Augmentation Strategycited by 54
  9. 2020What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical Perspectivecited by 35
  10. 2020An Empirical Evaluation on Robustness and Uncertainty of Regularization Methodscited by 28
  11. 2022Class-Specific Semantic Reconstruction for Open Set Recognitioncited by 28
  12. 2023Mutual Information-Based Temporal Difference Learning for Human Pose Estimation in Videocited by 17
  13. 2023The Devil is in the Wrongly-classified Samples: Towards Unified Open-set Recognitioncited by 16
  14. 2022Augmentation-Aware Self-Supervision for Data-Efficient GAN Trainingcited by 7
  15. 2022Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learningcited by 6
  16. 2019Class-Conditional Domain Adaptation on Semantic Segmentationcited by 4
  17. 2020Improved Robustness to Open Set Inputs via Tempered Mixupcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025UCDSC: Open Set UnCertainty aware Deep Simplex Classifier for Medical Image Datasetscited by 2
  2. 2025Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learningcited by 0
  3. 2025Boosting Open Set Recognition Performance through Modulated Representation Learningcited by 0
  4. 2026Beyond the Simplex: Balanced Prototype Geometry for Scorer-Agnostic Open-Set Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Preserving Angles Improves Feature Distillation
  2. 2022Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation
  3. 2026Vector-Quantized Soft Label Compression for Dataset Distillation
  4. 2026Beyond Dataset Distillation: Lossless Dataset Concentration via Diffusion-Assisted Distribution Alignment
  5. 2024Query-Based Knowledge Sharing for Open-Vocabulary Multi-Label Classification
  6. 2024Extract More from Less: Efficient Fine-Grained Visual Recognition in Low-Data Regimes
  7. 2022Teaching What You Should Teach: A Data-Based Distillation Method
  8. 2021Teacher-Explorer-Student Learning: A Novel Learning Method for Open Set Recognition