# Direct Differentiable Augmentation Search

arXiv: https://arxiv.org/abs/2104.04282 · HTML: https://mlsanity.com/paper/2104.04282/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 48 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring](https://mlsanity.com/paper/1911.09785/) (2019)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [Adversarial Examples Improve Image Recognition](https://mlsanity.com/paper/1911.09665/) (2019)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules](https://mlsanity.com/paper/1905.05393/) (2019)
- [Learning Data Augmentation Strategies for Object Detection](https://mlsanity.com/paper/1906.11172/) (2019)
- [Adversarial AutoAugment](https://mlsanity.com/paper/1912.11188/) (2019)
- [Backpropagation through the Void: Optimizing control variates for black-box gradient estimation](https://mlsanity.com/paper/1711.00123/) (2017)
- [Data Augmentation by Pairing Samples for Images Classification](https://mlsanity.com/paper/1801.02929/) (2018)
- [Faster AutoAugment: Learning Augmentation Strategies using Backpropagation](https://mlsanity.com/paper/1911.06987/) (2019)
- [Shake-Shake regularization](https://mlsanity.com/paper/1705.07485/) (2017)
- [PointAugment: an Auto-Augmentation Framework for Point Cloud Classification](https://mlsanity.com/paper/2002.10876/) (2020)

## Built on by (most cited first, 28 of 28)

- [Learning to Transform Dynamically for Better Adversarial Transferability](https://mlsanity.com/paper/2405.14077/) (2024)
- [Deep AutoAugment](https://mlsanity.com/paper/2203.06172/) (2022)
- [Automatic Data Augmentation via Invariance-Constrained Learning](https://mlsanity.com/paper/2209.15031/) (2022)
- [Automated data processing and feature engineering for deep learning and big data applications: a survey](https://mlsanity.com/paper/2403.11395/) (2024)
- [Relaxed Equivariance via Multitask Learning](https://mlsanity.com/paper/2410.17878/) (2024)
- [RangeAugment: Efficient Online Augmentation with Range Learning](https://mlsanity.com/paper/2212.10553/) (2022)
- [No Free Lunch in Self Supervised Representation Learning](https://mlsanity.com/paper/2304.11718/) (2023)
- [AROID: Improving Adversarial Robustness Through Online Instance-Wise Data Augmentation](https://mlsanity.com/paper/2306.07197/) (2023)
- [SLACK: Stable Learning of Augmentations with Cold-start and KL regularization](https://mlsanity.com/paper/2306.09998/) (2023)
- [DIVA: Dataset Derivative of a Learning Task](https://mlsanity.com/paper/2111.09785/) (2021)
- [Universal Adaptive Data Augmentation](https://mlsanity.com/paper/2207.06658/) (2022)
- [Sample-aware RandAugment: Search-free Automatic Data Augmentation for Effective Image Recognition](https://mlsanity.com/paper/2508.08004/) (2025)
- [DAAS: Differentiable Architecture and Augmentation Policy Search](https://mlsanity.com/paper/2109.15273/) (2021)
- [Empirical Evaluation of Data Augmentations for Biobehavioral Time Series Data with Deep Learning](https://mlsanity.com/paper/2210.06701/) (2022)
- [FreeAugment: Data Augmentation Search Across All Degrees of Freedom](https://mlsanity.com/paper/2409.04820/) (2024)
- [LEAVES: Learning Views for Time-Series Biobehavioral Data in Contrastive Learning](https://mlsanity.com/paper/2210.07340/) (2022)
- [Use as Many Surrogates as You Want: Selective Ensemble Attack to Unleash Transferability without Sacrificing Resource Efficiency](https://mlsanity.com/paper/2505.12644/) (2025)
- [Fine-grained Token Allocation Via Operation Pruning for Efficient MLLMs](https://mlsanity.com/paper/2507.02909/) (2025)
- [Optimizing transformations for contrastive learning in a differentiable framework](https://mlsanity.com/paper/2207.13367/) (2022)
- [Learning Self-Regularized Adversarial Views for Self-Supervised Vision Transformers](https://mlsanity.com/paper/2210.08458/) (2022)
- [Data augmentation with automated machine learning: approaches and performance comparison with classical data augmentation methods](https://mlsanity.com/paper/2403.08352/) (2024)
- [Optimizing Data Augmentation through Bayesian Model Selection](https://mlsanity.com/paper/2505.21813/) (2025)
- [On-the-Fly Data Augmentation via Gradient-Guided and Sample-Aware Influence Estimation](https://mlsanity.com/paper/2510.00434/) (2025)
- [Compositional Adversarial Training for Robust Visual Watermarking](https://mlsanity.com/paper/2605.16720/) (2026)
- [SODA: Self-organizing data augmentation in deep neural networks -- Application to biomedical image segmentation tasks](https://mlsanity.com/paper/2202.03223/) (2022)
- [Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Study](https://mlsanity.com/paper/2305.13232/) (2023)
- [The geometry of invariant learning: an information-theoretic analysis of data augmentation and generalization](https://mlsanity.com/paper/2602.14423/) (2026)
- [Ctrl-A: Control-Driven Online Data Augmentation](https://mlsanity.com/paper/2603.21819/) (2026)

## Abstract (arXiv)

Data augmentation has been an indispensable tool to improve the performance of deep neural networks, however the augmentation can hardly transfer among different tasks and datasets. Consequently, a recent trend is to adopt AutoML technique to learn proper augmentation policy without extensive hand-crafted tuning. In this paper, we propose an efficient differentiable search algorithm called Direct Differentiable Augmentation Search (DDAS). It exploits meta-learning with one-step gradient update and continuous relaxation to the expected training loss for efficient search. Our DDAS can achieve efficient augmentation search without relying on approximations such as Gumbel Softmax or second order gradient approximation. To further reduce the adverse effect of improper augmentations, we organize the search space into a two level hierarchy, in which we first decide whether to apply augmentation, and then determine the specific augmentation policy. On standard image classification benchmarks, our DDAS achieves state-of-the-art performance and efficiency tradeoff while reducing the search cost dramatically, e.g. 0.15 GPU hours for CIFAR-10. In addition, we also use DDAS to search augmentation for object detection task and achieve comparable performance with AutoAugment, while being 1000x faster.