# Regularizing Deep Networks with Semantic Data Augmentation

arXiv: https://arxiv.org/abs/2007.10538 · HTML: https://mlsanity.com/paper/2007.10538/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2020.

## Built on by (most cited first, 30 of 73)

- [Prompt Distribution Learning](https://mlsanity.com/paper/2205.03340/) (2022)
- [Mutual Consistency Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2109.09960/) (2021)
- [A Comprehensive Survey of Image Augmentation Techniques for Deep Learning](https://mlsanity.com/paper/2205.01491/) (2022)
- [SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2204.08808/) (2022)
- [EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones](https://mlsanity.com/paper/2211.09703/) (2022)
- [Probabilistic Contrastive Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2403.06726/) (2024)
- [A comprehensive survey on deep active learning in medical image analysis](https://mlsanity.com/paper/2310.14230/) (2023)
- [Implicit Sample Extension for Unsupervised Person Re-Identification](https://mlsanity.com/paper/2204.06892/) (2022)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [Real-world Image Dehazing with Coherence-based Pseudo Labeling and Cooperative Unfolding Network](https://mlsanity.com/paper/2406.07966/) (2024)
- [Multi-view Adversarial Discriminator: Mine the Non-causal Factors for Object Detection in Unseen Domains](https://mlsanity.com/paper/2304.02950/) (2023)
- [Learning Representational Invariances for Data-Efficient Action Recognition](https://mlsanity.com/paper/2103.16565/) (2021)
- [You Only Cut Once: Boosting Data Augmentation with a Single Cut](https://mlsanity.com/paper/2201.12078/) (2022)
- [A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application](https://mlsanity.com/paper/2310.14277/) (2023)
- [kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies](https://mlsanity.com/paper/2404.09447/) (2024)
- [EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Training](https://mlsanity.com/paper/2405.08768/) (2024)
- [Towards better understanding and better generalization of few-shot classification in histology images with contrastive learning](https://mlsanity.com/paper/2202.09059/) (2022)
- [Dual Adaptive Transformations for Weakly Supervised Point Cloud Segmentation](https://mlsanity.com/paper/2207.09084/) (2022)
- [FedImpro: Measuring and Improving Client Update in Federated Learning](https://mlsanity.com/paper/2402.07011/) (2024)
- [Semantic Distribution-aware Contrastive Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2105.05013/) (2021)
- [FAKD: Feature Augmented Knowledge Distillation for Semantic Segmentation](https://mlsanity.com/paper/2208.14143/) (2022)
- [CAM-loss: Towards Learning Spatially Discriminative Feature Representations](https://mlsanity.com/paper/2109.01359/) (2021)
- [Semi-supervised Semantic Segmentation with Mutual Knowledge Distillation](https://mlsanity.com/paper/2208.11499/) (2022)
- [Semantic Data Augmentation based Distance Metric Learning for Domain Generalization](https://mlsanity.com/paper/2208.02803/) (2022)
- [Optimizing PatchCore for Few/many-shot Anomaly Detection](https://mlsanity.com/paper/2307.10792/) (2023)
- [Adversarial Dual-Student with Differentiable Spatial Warping for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02792/) (2022)
- [Exploring Temporally Dynamic Data Augmentation for Video Recognition](https://mlsanity.com/paper/2206.15015/) (2022)
- [Learning with Partial Labels from Semi-supervised Perspective](https://mlsanity.com/paper/2211.13655/) (2022)
- [PASS++: A Dual Bias Reduction Framework for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2407.14029/) (2024)
- [Sill-Net: Feature Augmentation with Separated Illumination Representation](https://mlsanity.com/paper/2102.03539/) (2021)

## Abstract (arXiv)

Data augmentation is widely known as a simple yet surprisingly effective technique for regularizing deep networks. Conventional data augmentation schemes, e.g., flipping, translation or rotation, are low-level, data-independent and class-agnostic operations, leading to limited diversity for augmented samples. To this end, we propose a novel semantic data augmentation algorithm to complement traditional approaches. The proposed method is inspired by the intriguing property that deep networks are effective in learning linearized features, i.e., certain directions in the deep feature space correspond to meaningful semantic transformations, e.g., changing the background or view angle of an object. Based on this observation, translating training samples along many such directions in the feature space can effectively augment the dataset for more diversity. To implement this idea, we first introduce a sampling based method to obtain semantically meaningful directions efficiently. Then, an upper bound of the expected cross-entropy (CE) loss on the augmented training set is derived by assuming the number of augmented samples goes to infinity, yielding a highly efficient algorithm. In fact, we show that the proposed implicit semantic data augmentation (ISDA) algorithm amounts to minimizing a novel robust CE loss, which adds minimal extra computational cost to a normal training procedure. In addition to supervised learning, ISDA can be applied to semi-supervised learning tasks under the consistency regularization framework, where ISDA amounts to minimizing the upper bound of the expected KL-divergence between the augmented features and the original features. Although being simple, ISDA consistently improves the generalization performance of popular deep models (e.g., ResNets and DenseNets) on a variety of datasets, i.e., CIFAR-10, CIFAR-100, SVHN, ImageNet, and Cityscapes.