# Circumventing Outliers of AutoAugment with Knowledge Distillation

arXiv: https://arxiv.org/abs/2003.11342 · HTML: https://mlsanity.com/paper/2003.11342/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2020.

## Built on by (most cited first, 29 of 29)

- [Characterizing signal propagation to close the performance gap in unnormalized ResNets](https://mlsanity.com/paper/2101.08692/) (2021)
- [Unified Visual Transformer Compression](https://mlsanity.com/paper/2203.08243/) (2022)
- [TeachAugment: Data Augmentation Optimization Using Teacher Knowledge](https://mlsanity.com/paper/2202.12513/) (2022)
- [Comprehensive Survey of Model Compression and Speed up for Vision Transformers](https://mlsanity.com/paper/2404.10407/) (2024)
- [A Survey of Automated Data Augmentation Algorithms for Deep Learning-based Image Classification Tasks](https://mlsanity.com/paper/2206.06544/) (2022)
- [Feature Perturbation Augmentation for Reliable Evaluation of Importance Estimators in Neural Networks](https://mlsanity.com/paper/2303.01538/) (2023)
- [Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results](https://mlsanity.com/paper/2204.03475/) (2022)
- [Co-training 2L Submodels for Visual Recognition](https://mlsanity.com/paper/2212.04884/) (2022)
- [BasisNet: Two-stage Model Synthesis for Efficient Inference](https://mlsanity.com/paper/2105.03014/) (2021)
- [DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture](https://mlsanity.com/paper/2109.05765/) (2021)
- [Domain Generalization by Rejecting Extreme Augmentations](https://mlsanity.com/paper/2310.06670/) (2023)
- [Sample-aware RandAugment: Search-free Automatic Data Augmentation for Effective Image Recognition](https://mlsanity.com/paper/2508.08004/) (2025)
- [ReSmooth: Detecting and Utilizing OOD Samples when Training with Data Augmentation](https://mlsanity.com/paper/2205.12606/) (2022)
- [HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification](https://mlsanity.com/paper/2407.07516/) (2024)
- [Unsupervised Domain Adaptation on Person Re-Identification via Dual-level Asymmetric Mutual Learning](https://mlsanity.com/paper/2301.12439/) (2023)
- [Exploring the Diversity and Invariance in Yourself for Visual Pre-Training Task](https://mlsanity.com/paper/2106.00537/) (2021)
- [ParticleAugment: Sampling-Based Data Augmentation](https://mlsanity.com/paper/2106.08693/) (2021)
- [Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer](https://mlsanity.com/paper/2208.01161/) (2022)
- [Learning Self-Regularized Adversarial Views for Self-Supervised Vision Transformers](https://mlsanity.com/paper/2210.08458/) (2022)
- [DualAug: Exploiting Additional Heavy Augmentation with OOD Data Rejection](https://mlsanity.com/paper/2310.08139/) (2023)
- [Data augmentation with automated machine learning: approaches and performance comparison with classical data augmentation methods](https://mlsanity.com/paper/2403.08352/) (2024)
- [Low-Resolution Chest X-ray Classification via Knowledge Distillation and Multi-task Learning](https://mlsanity.com/paper/2405.13370/) (2024)
- [InAugment: Improving Classifiers via Internal Augmentation](https://mlsanity.com/paper/2104.03843/) (2021)
- [Source Domain Subset Sampling for Semi-Supervised Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/2205.00312/) (2022)
- [A General Multiple Data Augmentation Based Framework for Training Deep Neural Networks](https://mlsanity.com/paper/2205.14606/) (2022)
- [Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Study](https://mlsanity.com/paper/2305.13232/) (2023)
- [Preserving Angles Improves Feature Distillation](https://mlsanity.com/paper/2411.15239/) (2024)
- [Accelerating Vision Transformers on Brain Processing Unit](https://mlsanity.com/paper/2602.06300/) (2026)
- [Sliding-Window Reordering with Overlap Averaging: A Simple Time-Domain Augmentation for Multivariate Forecasting](https://mlsanity.com/paper/2604.09067/) (2026)

## Abstract (arXiv)

AutoAugment has been a powerful algorithm that improves the accuracy of many vision tasks, yet it is sensitive to the operator space as well as hyper-parameters, and an improper setting may degenerate network optimization. This paper delves deep into the working mechanism, and reveals that AutoAugment may remove part of discriminative information from the training image and so insisting on the ground-truth label is no longer the best option. To relieve the inaccuracy of supervision, we make use of knowledge distillation that refers to the output of a teacher model to guide network training. Experiments are performed in standard image classification benchmarks, and demonstrate the effectiveness of our approach in suppressing noise of data augmentation and stabilizing training. Upon the cooperation of knowledge distillation and AutoAugment, we claim the new state-of-the-art on ImageNet classification with a top-1 accuracy of 85.8%.