# Revealing the Two Sides of Data Augmentation: An Asymmetric Distillation-based Win-Win Solution for Open-Set Recognition

arXiv: https://arxiv.org/abs/2404.19527 · HTML: https://mlsanity.com/paper/2404.19527/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 28 Apr 2024.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [Adversarial Reciprocal Points Learning for Open Set Recognition](https://mlsanity.com/paper/2103.00953/) (2021)
- [Dataset Augmentation in Feature Space](https://mlsanity.com/paper/1702.05538/) (2017)
- [Image Data Augmentation for Deep Learning: A Survey](https://mlsanity.com/paper/2204.08610/) (2022)
- [The Effects of Regularization and Data Augmentation are Class Dependent](https://mlsanity.com/paper/2204.03632/) (2022)
- [Smart Augmentation - Learning an Optimal Data Augmentation Strategy](https://mlsanity.com/paper/1703.08383/) (2017)
- [What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical Perspective](https://mlsanity.com/paper/2012.02909/) (2020)
- [An Empirical Evaluation on Robustness and Uncertainty of Regularization Methods](https://mlsanity.com/paper/2003.03879/) (2020)
- [Class-Specific Semantic Reconstruction for Open Set Recognition](https://mlsanity.com/paper/2207.02158/) (2022)
- [Mutual Information-Based Temporal Difference Learning for Human Pose Estimation in Video](https://mlsanity.com/paper/2303.08475/) (2023)
- [The Devil is in the Wrongly-classified Samples: Towards Unified Open-set Recognition](https://mlsanity.com/paper/2302.04002/) (2023)
- [Augmentation-Aware Self-Supervision for Data-Efficient GAN Training](https://mlsanity.com/paper/2205.15677/) (2022)
- [Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning](https://mlsanity.com/paper/2212.08663/) (2022)
- [Understanding the Role of Mixup in Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2211.03946/) (2022)
- [Class-Conditional Domain Adaptation on Semantic Segmentation](https://mlsanity.com/paper/1911.11981/) (2019)

## Built on by (most cited first, 4 of 4)

- [UCDSC: Open Set UnCertainty aware Deep Simplex Classifier for Medical Image Datasets](https://mlsanity.com/paper/2511.08196/) (2025)
- [Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learning](https://mlsanity.com/paper/2504.11930/) (2025)
- [Boosting Open Set Recognition Performance through Modulated Representation Learning](https://mlsanity.com/paper/2505.18137/) (2025)
- [Beyond the Simplex: Balanced Prototype Geometry for Scorer-Agnostic Open-Set Recognition](https://mlsanity.com/paper/2606.01883/) (2026)

## Abstract (arXiv)

In this paper, we reveal the two sides of data augmentation: enhancements in closed-set recognition correlate with a significant decrease in open-set recognition. Through empirical investigation, we find that multi-sample-based augmentations would contribute to reducing feature discrimination, thereby diminishing the open-set criteria. Although knowledge distillation could impair the feature via imitation, the mixed feature with ambiguous semantics hinders the distillation. To this end, we propose an asymmetric distillation framework by feeding teacher model extra raw data to enlarge the benefit of teacher. Moreover, a joint mutual information loss and a selective relabel strategy are utilized to alleviate the influence of hard mixed samples. Our method successfully mitigates the decline in open-set and outperforms SOTAs by 2%~3% AUROC on the Tiny-ImageNet dataset and experiments on large-scale dataset ImageNet-21K demonstrate the generalization of our method.