# DADA: Differentiable Automatic Data Augmentation

arXiv: https://arxiv.org/abs/2003.03780 · HTML: https://mlsanity.com/paper/2003.03780/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2020.

## Built on by (most cited first, 30 of 57)

- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models](https://mlsanity.com/paper/2405.14881/) (2024)
- [Time Series Contrastive Learning with Information-Aware Augmentations](https://mlsanity.com/paper/2303.11911/) (2023)
- [Learning Partial Equivariances from Data](https://mlsanity.com/paper/2110.10211/) (2021)
- [TeachAugment: Data Augmentation Optimization Using Teacher Knowledge](https://mlsanity.com/paper/2202.12513/) (2022)
- [Automated Progressive Learning for Efficient Training of Vision Transformers](https://mlsanity.com/paper/2203.14509/) (2022)
- [Direct Differentiable Augmentation Search](https://mlsanity.com/paper/2104.04282/) (2021)
- [Scale-aware Automatic Augmentation for Object Detection](https://mlsanity.com/paper/2103.17220/) (2021)
- [Image Data Augmentation Approaches: A Comprehensive Survey and Future directions](https://mlsanity.com/paper/2301.02830/) (2023)
- [CADDA: Class-wise Automatic Differentiable Data Augmentation for EEG Signals](https://mlsanity.com/paper/2106.13695/) (2021)
- [Automatic Data Augmentation via Invariance-Constrained Learning](https://mlsanity.com/paper/2209.15031/) (2022)
- [A Lower Bound and a Near-Optimal Algorithm for Bilevel Empirical Risk Minimization](https://mlsanity.com/paper/2302.08766/) (2023)
- [Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification](https://mlsanity.com/paper/2109.00523/) (2021)
- [Learning Instance-Specific Augmentations by Capturing Local Invariances](https://mlsanity.com/paper/2206.00051/) (2022)
- [AutoDO: Robust AutoAugment for Biased Data with Label Noise via Scalable Probabilistic Implicit Differentiation](https://mlsanity.com/paper/2103.05863/) (2021)
- [An overview of mixing augmentation methods and augmentation strategies](https://mlsanity.com/paper/2107.09887/) (2021)
- [Automated data processing and feature engineering for deep learning and big data applications: a survey](https://mlsanity.com/paper/2403.11395/) (2024)
- [GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editing](https://mlsanity.com/paper/2412.02366/) (2024)
- [Learning Augmentation Distributions using Transformed Risk Minimization](https://mlsanity.com/paper/2111.08190/) (2021)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [Masked Autoencoders are Robust Data Augmentors](https://mlsanity.com/paper/2206.04846/) (2022)
- [When to Learn What: Model-Adaptive Data Augmentation Curriculum](https://mlsanity.com/paper/2309.04747/) (2023)
- [Understanding the Detrimental Class-level Effects of Data Augmentation](https://mlsanity.com/paper/2401.01764/) (2023)
- [EntAugment: Entropy-Driven Adaptive Data Augmentation Framework for Image Classification](https://mlsanity.com/paper/2409.06290/) (2024)
- [A Survey of Automated Data Augmentation Algorithms for Deep Learning-based Image Classification Tasks](https://mlsanity.com/paper/2206.06544/) (2022)
- [SelectAugment: Hierarchical Deterministic Sample Selection for Data Augmentation](https://mlsanity.com/paper/2112.02862/) (2021)
- [AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation](https://mlsanity.com/paper/2203.06558/) (2022)
- [Dense Vision Transformer Compression with Few Samples](https://mlsanity.com/paper/2403.18708/) (2024)
- [Local Patch AutoAugment with Multi-Agent Collaboration](https://mlsanity.com/paper/2103.11099/) (2021)
- [Hyperparameter Optimization through Neural Network Partitioning](https://mlsanity.com/paper/2304.14766/) (2023)

## Abstract (arXiv)

Data augmentation (DA) techniques aim to increase data variability, and thus train deep networks with better generalisation. The pioneering AutoAugment automated the search for optimal DA policies with reinforcement learning. However, AutoAugment is extremely computationally expensive, limiting its wide applicability. Followup works such as Population Based Augmentation (PBA) and Fast AutoAugment improved efficiency, but their optimization speed remains a bottleneck. In this paper, we propose Differentiable Automatic Data Augmentation (DADA) which dramatically reduces the cost. DADA relaxes the discrete DA policy selection to a differentiable optimization problem via Gumbel-Softmax. In addition, we introduce an unbiased gradient estimator, RELAX, leading to an efficient and effective one-pass optimization strategy to learn an efficient and accurate DA policy. We conduct extensive experiments on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets. Furthermore, we demonstrate the value of Auto DA in pre-training for downstream detection problems. Results show our DADA is at least one order of magnitude faster than the state-of-the-art while achieving very comparable accuracy. The code is available at https://github.com/VDIGPKU/DADA.