# Implicit Semantic Data Augmentation for Deep Networks

arXiv: https://arxiv.org/abs/1909.12220 · HTML: https://mlsanity.com/paper/1909.12220/

## Key facts

- Cited by 106 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2019.

## Built on by (most cited first, 30 of 106)

- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [Free Lunch for Few-shot Learning: Distribution Calibration](https://mlsanity.com/paper/2101.06395/) (2021)
- [Uncertainty Modeling for Out-of-Distribution Generalization](https://mlsanity.com/paper/2202.03958/) (2022)
- [The Majority Can Help The Minority: Context-rich Minority Oversampling for Long-tailed Classification](https://mlsanity.com/paper/2112.00412/) (2021)
- [MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2103.12579/) (2021)
- [Source-Free Domain Adaptation via Distribution Estimation](https://mlsanity.com/paper/2204.11257/) (2022)
- [Adaptive Focus for Efficient Video Recognition](https://mlsanity.com/paper/2105.03245/) (2021)
- [FASA: Feature Augmentation and Sampling Adaptation for Long-Tailed Instance Segmentation](https://mlsanity.com/paper/2102.12867/) (2021)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [Transferable Semantic Augmentation for Domain Adaptation](https://mlsanity.com/paper/2103.12562/) (2021)
- [RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression](https://mlsanity.com/paper/2205.15236/) (2022)
- [Spectral Feature Augmentation for Graph Contrastive Learning and Beyond](https://mlsanity.com/paper/2212.01026/) (2022)
- [EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones](https://mlsanity.com/paper/2211.09703/) (2022)
- [Glance and Focus Networks for Dynamic Visual Recognition](https://mlsanity.com/paper/2201.03014/) (2022)
- [COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning](https://mlsanity.com/paper/2206.04726/) (2022)
- [AutoDropout: Learning Dropout Patterns to Regularize Deep Networks](https://mlsanity.com/paper/2101.01761/) (2021)
- [Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering](https://mlsanity.com/paper/2404.10193/) (2024)
- [SplineCam: Exact Visualization and Characterization of Deep Network Geometry and Decision Boundaries](https://mlsanity.com/paper/2302.12828/) (2023)
- [A Systematic Review on Long-Tailed Learning](https://mlsanity.com/paper/2408.00483/) (2024)
- [Towards Understanding the Generative Capability of Adversarially Robust Classifiers](https://mlsanity.com/paper/2108.09093/) (2021)
- [Imagine by Reasoning: A Reasoning-Based Implicit Semantic Data Augmentation for Long-Tailed Classification](https://mlsanity.com/paper/2112.07928/) (2021)
- [GSmooth: Certified Robustness against Semantic Transformations via Generalized Randomized Smoothing](https://mlsanity.com/paper/2206.04310/) (2022)
- [StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis](https://mlsanity.com/paper/2312.10741/) (2023)
- [SF(DA)2: Source-free Domain Adaptation Through the Lens of Data Augmentation](https://mlsanity.com/paper/2403.10834/) (2024)
- [AdaFocusV3: On Unified Spatial-temporal Dynamic Video Recognition](https://mlsanity.com/paper/2209.13465/) (2022)
- [PIDray: A Large-scale X-ray Benchmark for Real-World Prohibited Item Detection](https://mlsanity.com/paper/2211.10763/) (2022)
- [Fine-grained Recognition with Learnable Semantic Data Augmentation](https://mlsanity.com/paper/2309.00399/) (2023)
- [Towards better understanding and better generalization of few-shot classification in histology images with contrastive learning](https://mlsanity.com/paper/2202.09059/) (2022)

## Abstract (arXiv)

In this paper, we propose a novel implicit semantic data augmentation (ISDA) approach to complement traditional augmentation techniques like flipping, translation or rotation. Our work is motivated by the intriguing property that deep networks are surprisingly good at linearizing features, such that certain directions in the deep feature space correspond to meaningful semantic transformations, e.g., adding sunglasses or changing backgrounds. As a consequence, translating training samples along many semantic directions in the feature space can effectively augment the dataset to improve generalization. To implement this idea effectively and efficiently, we first perform an online estimate of the covariance matrix of deep features for each class, which captures the intra-class semantic variations. Then random vectors are drawn from a zero-mean normal distribution with the estimated covariance to augment the training data in that class. Importantly, instead of augmenting the samples explicitly, we can directly minimize an upper bound of the expected cross-entropy (CE) loss on the augmented training set, leading to a highly efficient algorithm. In fact, we show that the proposed ISDA amounts to minimizing a novel robust CE loss, which adds negligible extra computational cost to a normal training procedure. Although being simple, ISDA consistently improves the generalization performance of popular deep models (ResNets and DenseNets) on a variety of datasets, e.g., CIFAR-10, CIFAR-100 and ImageNet. Code for reproducing our results is available at https://github.com/blackfeather-wang/ISDA-for-Deep-Networks.