# SLACK: Stable Learning of Augmentations with Cold-start and KL regularization

arXiv: https://arxiv.org/abs/2306.09998 · HTML: https://mlsanity.com/paper/2306.09998/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules](https://mlsanity.com/paper/1905.05393/) (2019)
- [Adversarial AutoAugment](https://mlsanity.com/paper/1912.11188/) (2019)
- [Backpropagation through the Void: Optimizing control variates for black-box gradient estimation](https://mlsanity.com/paper/1711.00123/) (2017)
- [Faster AutoAugment: Learning Augmentation Strategies using Backpropagation](https://mlsanity.com/paper/1911.06987/) (2019)
- [DADA: Differentiable Automatic Data Augmentation](https://mlsanity.com/paper/2003.03780/) (2020)
- [Online Hyper-parameter Learning for Auto-Augmentation Strategy](https://mlsanity.com/paper/1905.07373/) (2019)
- [UniformAugment: A Search-free Probabilistic Data Augmentation Approach](https://mlsanity.com/paper/2003.14348/) (2020)
- [TeachAugment: Data Augmentation Optimization Using Teacher Knowledge](https://mlsanity.com/paper/2202.12513/) (2022)
- [Direct Differentiable Augmentation Search](https://mlsanity.com/paper/2104.04282/) (2021)
- [Non-Convex Bilevel Games with Critical Point Selection Maps](https://mlsanity.com/paper/2207.04888/) (2022)
- [Meta Approach to Data Augmentation Optimization](https://mlsanity.com/paper/2006.07965/) (2020)
- [DAAS: Differentiable Architecture and Augmentation Policy Search](https://mlsanity.com/paper/2109.15273/) (2021)

## Built on by (most cited first, 6 of 6)

- [Functional Bilevel Optimization for Machine Learning](https://mlsanity.com/paper/2403.20233/) (2024)
- [Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification](https://mlsanity.com/paper/2403.01944/) (2024)
- [Sample-aware RandAugment: Search-free Automatic Data Augmentation for Effective Image Recognition](https://mlsanity.com/paper/2508.08004/) (2025)
- [FreeAugment: Data Augmentation Search Across All Degrees of Freedom](https://mlsanity.com/paper/2409.04820/) (2024)
- [On-the-Fly Data Augmentation via Gradient-Guided and Sample-Aware Influence Estimation](https://mlsanity.com/paper/2510.00434/) (2025)
- [Class-specific Data Augmentation for Plant Stress Classification](https://mlsanity.com/paper/2406.13081/) (2024)

## Abstract (arXiv)

Data augmentation is known to improve the generalization capabilities of neural networks, provided that the set of transformations is chosen with care, a selection often performed manually. Automatic data augmentation aims at automating this process. However, most recent approaches still rely on some prior information; they start from a small pool of manually-selected default transformations that are either used to pretrain the network or forced to be part of the policy learned by the automatic data augmentation algorithm. In this paper, we propose to directly learn the augmentation policy without leveraging such prior knowledge. The resulting bilevel optimization problem becomes more challenging due to the larger search space and the inherent instability of bilevel optimization algorithms. To mitigate these issues (i) we follow a successive cold-start strategy with a Kullback-Leibler regularization, and (ii) we parameterize magnitudes as continuous distributions. Our approach leads to competitive results on standard benchmarks despite a more challenging setting, and generalizes beyond natural images.