# When to Learn What: Model-Adaptive Data Augmentation Curriculum

arXiv: https://arxiv.org/abs/2309.04747 · HTML: https://mlsanity.com/paper/2309.04747/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 51 references that are in the map.
- First posted on arXiv 9 Sep 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [Training Very Deep Networks](https://mlsanity.com/paper/1507.06228/) (2015)
- [Deep Pyramidal Residual Networks](https://mlsanity.com/paper/1610.02915/) (2016)
- [Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules](https://mlsanity.com/paper/1905.05393/) (2019)
- [Adversarial AutoAugment](https://mlsanity.com/paper/1912.11188/) (2019)
- [Faster AutoAugment: Learning Augmentation Strategies using Backpropagation](https://mlsanity.com/paper/1911.06987/) (2019)
- [Shake-Shake regularization](https://mlsanity.com/paper/1705.07485/) (2017)
- [The Effects of Regularization and Data Augmentation are Class Dependent](https://mlsanity.com/paper/2204.03632/) (2022)
- [PointAugment: an Auto-Augmentation Framework for Point Cloud Classification](https://mlsanity.com/paper/2002.10876/) (2020)
- [Data augmentation instead of explicit regularization](https://mlsanity.com/paper/1806.03852/) (2018)
- [DADA: Differentiable Automatic Data Augmentation](https://mlsanity.com/paper/2003.03780/) (2020)
- [Between-class Learning for Image Classification](https://mlsanity.com/paper/1711.10284/) (2017)
- [ShakeDrop Regularization for Deep Residual Learning](https://mlsanity.com/paper/1802.02375/) (2018)
- [Online Hyper-parameter Learning for Auto-Augmentation Strategy](https://mlsanity.com/paper/1905.07373/) (2019)

## Built on by (most cited first, 13 of 13)

- [Fake It till You Make It: Curricular Dynamic Forgery Augmentations towards General Deepfake Detection](https://mlsanity.com/paper/2409.14444/) (2024)
- [EntAugment: Entropy-Driven Adaptive Data Augmentation Framework for Image Classification](https://mlsanity.com/paper/2409.06290/) (2024)
- [AdaAugment: A Tuning-Free and Adaptive Approach to Enhance Data Augmentation](https://mlsanity.com/paper/2405.11467/) (2024)
- [Diffusion Curriculum: Synthetic-to-Real Data Curriculum via Image-Guided Diffusion](https://mlsanity.com/paper/2410.13674/) (2024)
- [When Dynamic Data Selection Meets Data Augmentation](https://mlsanity.com/paper/2505.03809/) (2025)
- [Sample-aware RandAugment: Search-free Automatic Data Augmentation for Effective Image Recognition](https://mlsanity.com/paper/2508.08004/) (2025)
- [Learning to See Through a Baby's Eyes: Early Visual Diets Enable Robust Visual Intelligence in Humans and Machines](https://mlsanity.com/paper/2511.14440/) (2025)
- [Long-Tailed Backdoor Attack Using Dynamic Data Augmentation Operations](https://mlsanity.com/paper/2410.12955/) (2024)
- [Enhancing Small-Scale Dataset Expansion with Triplet-Connection-based Sample Re-Weighting](https://mlsanity.com/paper/2508.07723/) (2025)
- [On-the-Fly Data Augmentation via Gradient-Guided and Sample-Aware Influence Estimation](https://mlsanity.com/paper/2510.00434/) (2025)
- [Optimal Layer Selection for Latent Data Augmentation](https://mlsanity.com/paper/2408.13426/) (2024)
- [History Is Not Enough: An Adaptive Dataflow System for Financial Time-Series Synthesis](https://mlsanity.com/paper/2601.10143/) (2026)
- [Learning-State-Aware Dynamic Generative Data Augmentation on Small-Scale Datasets](https://mlsanity.com/paper/2608.18907/) (2026)

## Abstract (arXiv)

Data augmentation (DA) is widely used to improve the generalization of neural networks by enforcing the invariances and symmetries to pre-defined transformations applied to input data. However, a fixed augmentation policy may have different effects on each sample in different training stages but existing approaches cannot adjust the policy to be adaptive to each sample and the training model. In this paper, we propose Model Adaptive Data Augmentation (MADAug) that jointly trains an augmentation policy network to teach the model when to learn what. Unlike previous work, MADAug selects augmentation operators for each input image by a model-adaptive policy varying between training stages, producing a data augmentation curriculum optimized for better generalization. In MADAug, we train the policy through a bi-level optimization scheme, which aims to minimize a validation-set loss of a model trained using the policy-produced data augmentations. We conduct an extensive evaluation of MADAug on multiple image classification tasks and network architectures with thorough comparisons to existing DA approaches. MADAug outperforms or is on par with other baselines and exhibits better fairness: it brings improvement to all classes and more to the difficult ones. Moreover, MADAug learned policy shows better performance when transferred to fine-grained datasets. In addition, the auto-optimized policy in MADAug gradually introduces increasing perturbations and naturally forms an easy-to-hard curriculum.