# Negative Data Augmentation

arXiv: https://arxiv.org/abs/2102.05113 · HTML: https://mlsanity.com/paper/2102.05113/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 30 of 30)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [Video Representation Learning by Dense Predictive Coding](https://mlsanity.com/paper/1909.04656/) (2019)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations](https://mlsanity.com/paper/1807.01697/) (2018)
- [Bias and Generalization in Deep Generative Models: An Empirical Study](https://mlsanity.com/paper/1811.03259/) (2018)
- [A critical analysis of self-supervision, or what we can learn from a single image](https://mlsanity.com/paper/1904.13132/) (2019)
- [Adversarial Contrastive Estimation](https://mlsanity.com/paper/1805.03642/) (2018)
- [Unsupervised Adversarial Invariance](https://mlsanity.com/paper/1809.10083/) (2018)
- [Actively Avoiding Nonsense in Generative Models](https://mlsanity.com/paper/1802.07229/) (2018)

## Built on by (most cited first, 30 of 46)

- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?](https://mlsanity.com/paper/2104.09425/) (2021)
- [Full-Spectrum Out-of-Distribution Detection](https://mlsanity.com/paper/2204.05306/) (2022)
- [Harnessing Out-Of-Distribution Examples via Augmenting Content and Style](https://mlsanity.com/paper/2207.03162/) (2022)
- [Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation](https://mlsanity.com/paper/2110.07858/) (2021)
- [State-of-the-Art in the Architecture, Methods and Applications of StyleGAN](https://mlsanity.com/paper/2202.14020/) (2022)
- [A Practical Contrastive Learning Framework for Single-Image Super-Resolution](https://mlsanity.com/paper/2111.13924/) (2021)
- [The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective](https://mlsanity.com/paper/2210.05021/) (2022)
- [SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps](https://mlsanity.com/paper/2201.05120/) (2022)
- [PartMix: Regularization Strategy to Learn Part Discovery for Visible-Infrared Person Re-identification](https://mlsanity.com/paper/2304.01537/) (2023)
- [Data Redaction from Pre-trained GANs](https://mlsanity.com/paper/2206.14389/) (2022)
- [Nuisances via Negativa: Adjusting for Spurious Correlations via Data Augmentation](https://mlsanity.com/paper/2210.01302/) (2022)
- [Does Negative Sampling Matter? A Review with Insights into its Theory and Applications](https://mlsanity.com/paper/2402.17238/) (2024)
- [Learning Equi-angular Representations for Online Continual Learning](https://mlsanity.com/paper/2404.01628/) (2024)
- [Anomaly Detection by Leveraging Incomplete Anomalous Knowledge with Anomaly-Aware Bidirectional GANs](https://mlsanity.com/paper/2204.13335/) (2022)
- [Constraining Generative Models for Engineering Design with Negative Data](https://mlsanity.com/paper/2306.15166/) (2023)
- [Novelty Detection via Contrastive Learning with Negative Data Augmentation](https://mlsanity.com/paper/2106.09958/) (2021)
- [Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusion](https://mlsanity.com/paper/2312.03475/) (2023)
- [Understanding Hyperbolic Metric Learning through Hard Negative Sampling](https://mlsanity.com/paper/2404.15523/) (2024)
- [Recommendations on test datasets for evaluating AI solutions in pathology](https://mlsanity.com/paper/2204.14226/) (2022)
- [Data Redaction from Conditional Generative Models](https://mlsanity.com/paper/2305.11351/) (2023)
- [PatchGuard: Adversarially Robust Anomaly Detection and Localization through Vision Transformers and Pseudo Anomalies](https://mlsanity.com/paper/2506.09237/) (2025)
- [Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts](https://mlsanity.com/paper/2108.11926/) (2021)
- [Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration](https://mlsanity.com/paper/2112.08132/) (2021)
- [Don't be so negative! Score-based Generative Modeling with Oracle-assisted Guidance](https://mlsanity.com/paper/2307.16463/) (2023)
- [AE-StyleGAN: Improved Training of Style-Based Auto-Encoders](https://mlsanity.com/paper/2110.08718/) (2021)
- [AugCSE: Contrastive Sentence Embedding with Diverse Augmentations](https://mlsanity.com/paper/2210.13749/) (2022)
- [Model Inversion Attack against Transfer Learning: Inverting a Model without Accessing It](https://mlsanity.com/paper/2203.06570/) (2022)
- [Class-wise Thresholding for Robust Out-of-Distribution Detection](https://mlsanity.com/paper/2110.15292/) (2021)

## Abstract (arXiv)

Data augmentation is often used to enlarge datasets with synthetic samples generated in accordance with the underlying data distribution. To enable a wider range of augmentations, we explore negative data augmentation strategies (NDA)that intentionally create out-of-distribution samples. We show that such negative out-of-distribution samples provide information on the support of the data distribution, and can be leveraged for generative modeling and representation learning. We introduce a new GAN training objective where we use NDA as an additional source of synthetic data for the discriminator. We prove that under suitable conditions, optimizing the resulting objective still recovers the true data distribution but can directly bias the generator towards avoiding samples that lack the desired structure. Empirically, models trained with our method achieve improved conditional/unconditional image generation along with improved anomaly detection capabilities. Further, we incorporate the same negative data augmentation strategy in a contrastive learning framework for self-supervised representation learning on images and videos, achieving improved performance on downstream image classification, object detection, and action recognition tasks. These results suggest that prior knowledge on what does not constitute valid data is an effective form of weak supervision across a range of unsupervised learning tasks.