mlsanityData to 7 Oct 2026

Circumventing Outliers of AutoAugment with Knowledge Distillation

arXiv 2003.11342 ↗First posted 25 Mar 2020cs.CVcited by 29

Not on the map. Why?

Abstract (arXiv)

AutoAugment has been a powerful algorithm that improves the accuracy of many vision tasks, yet it is sensitive to the operator space as well as hyper-parameters, and an improper setting may degenerate network optimization. This paper delves deep into the working mechanism, and reveals that AutoAugment may remove part of discriminative information from the training image and so insisting on the ground-truth label is no longer the best option. To relieve the inaccuracy of supervision, we make use of knowledge distillation that refers to the output of a teacher model to guide network training. Experiments are performed in standard image classification benchmarks, and demonstrate the effectiveness of our approach in suppressing noise of data augmentation and stabilizing training. Upon the cooperation of knowledge distillation and AutoAugment, we claim the new state-of-the-art on ImageNet classification with a top-1 accuracy of 85.8%.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 10; 2023: 5; 2024: 5; 2025: 1; 2026: 2.

29 papers.

Sort:

  1. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  2. 2022Unified Visual Transformer Compressioncited by 79
  3. 2022TeachAugment: Data Augmentation Optimization Using Teacher Knowledgecited by 37
  4. 2024Comprehensive Survey of Model Compression and Speed up for Vision Transformerscited by 14
  5. 2022A Survey of Automated Data Augmentation Algorithms for Deep Learning-based Image Classification Taskscited by 12
  6. 2023Feature Perturbation Augmentation for Reliable Evaluation of Importance Estimators in Neural Networkscited by 9
  7. 2022Solving ImageNet: a Unified Scheme for Training any Backbone to Top Resultscited by 8
  8. 2022Co-training 2L Submodels for Visual Recognitioncited by 8
  9. 2021BasisNet: Two-stage Model Synthesis for Efficient Inferencecited by 7
  10. 2021DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecturecited by 6
  11. 2023Domain Generalization by Rejecting Extreme Augmentationscited by 5
  12. 2025Sample-aware RandAugment: Search-free Automatic Data Augmentation for Effective Image Recognitioncited by 5
17 more
  1. 2022ReSmooth: Detecting and Utilizing OOD Samples when Training with Data Augmentationcited by 3
  2. 2024HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classificationcited by 3
  3. 2023Unsupervised Domain Adaptation on Person Re-Identification via Dual-level Asymmetric Mutual Learningcited by 2
  4. 2021Exploring the Diversity and Invariance in Yourself for Visual Pre-Training Taskcited by 1
  5. 2021ParticleAugment: Sampling-Based Data Augmentationcited by 1
  6. 2022Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformercited by 1
  7. 2022Learning Self-Regularized Adversarial Views for Self-Supervised Vision Transformerscited by 1
  8. 2023DualAug: Exploiting Additional Heavy Augmentation with OOD Data Rejectioncited by 1
  9. 2024Data augmentation with automated machine learning: approaches and performance comparison with classical data augmentation methodscited by 1
  10. 2024Low-Resolution Chest X-ray Classification via Knowledge Distillation and Multi-task Learningcited by 1
  11. 2021InAugment: Improving Classifiers via Internal Augmentationcited by 0
  12. 2022Source Domain Subset Sampling for Semi-Supervised Domain Adaptation in Semantic Segmentationcited by 0
  13. 2022A General Multiple Data Augmentation Based Framework for Training Deep Neural Networkscited by 0
  14. 2023Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Studycited by 0
  15. 2024Preserving Angles Improves Feature Distillationcited by 0
  16. 2026Accelerating Vision Transformers on Brain Processing Unitcited by 0
  17. 2026Sliding-Window Reordering with Overlap Averaging: A Simple Time-Domain Augmentation for Multivariate Forecastingcited by 0