# Deep invariant networks with differentiable augmentation layers

arXiv: https://arxiv.org/abs/2202.02142 · HTML: https://mlsanity.com/paper/2202.02142/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 56 references that are in the map.
- First posted on arXiv 4 Feb 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [Kornia: an Open Source Differentiable Computer Vision Library for PyTorch](https://mlsanity.com/paper/1910.02190/) (2019)
- [Understanding and Robustifying Differentiable Architecture Search](https://mlsanity.com/paper/1909.09656/) (2019)
- [Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules](https://mlsanity.com/paper/1905.05393/) (2019)
- [Gradient Estimation Using Stochastic Computation Graphs](https://mlsanity.com/paper/1506.05254/) (2015)
- [Backpropagation through the Void: Optimizing control variates for black-box gradient estimation](https://mlsanity.com/paper/1711.00123/) (2017)
- [Stabilizing Differentiable Architecture Search via Perturbation-based Regularization](https://mlsanity.com/paper/2002.05283/) (2020)
- [Data Augmentation by Pairing Samples for Images Classification](https://mlsanity.com/paper/1801.02929/) (2018)
- [Faster AutoAugment: Learning Augmentation Strategies using Backpropagation](https://mlsanity.com/paper/1911.06987/) (2019)
- [XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging](https://mlsanity.com/paper/2007.05492/) (2020)
- [Meta-Learning Symmetries by Reparameterization](https://mlsanity.com/paper/2007.02933/) (2020)
- [A deep learning architecture for temporal sleep stage classification using multivariate and multimodal time series](https://mlsanity.com/paper/1707.03321/) (2017)
- [DADA: Differentiable Automatic Data Augmentation](https://mlsanity.com/paper/2003.03780/) (2020)
- [Learning Invariances in Neural Networks](https://mlsanity.com/paper/2010.11882/) (2020)
- [Learning Invariances using the Marginal Likelihood](https://mlsanity.com/paper/1808.05563/) (2018)
- [iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients](https://mlsanity.com/paper/2106.10784/) (2021)
- [Some Improvements on Deep Convolutional Neural Network Based Image Classification](https://mlsanity.com/paper/1312.5402/) (2013)
- [Grounding inductive biases in natural images:invariance stems from variations in data](https://mlsanity.com/paper/2106.05121/) (2021)
- [CADDA: Class-wise Automatic Differentiable Data Augmentation for EEG Signals](https://mlsanity.com/paper/2106.13695/) (2021)
- [Addressing Class Imbalance in Classification Problems of Noisy Signals by using Fourier Transform Surrogates](https://mlsanity.com/paper/1806.08675/) (2018)

## Built on by (most cited first, 6 of 6)

- [Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance](https://mlsanity.com/paper/2306.02866/) (2023)
- [Equivariant plug-and-play image reconstruction](https://mlsanity.com/paper/2312.01831/) (2023)
- [A Generative Model of Symmetry Transformations](https://mlsanity.com/paper/2403.01946/) (2024)
- [Follow the Energy, Find the Path: Riemannian Metrics from Energy-Based Models](https://mlsanity.com/paper/2505.18230/) (2025)
- [Dynamic Test-Time Augmentation via Differentiable Functions](https://mlsanity.com/paper/2212.04681/) (2022)
- [Review of Deep Representation Learning Techniques for Brain-Computer Interfaces and Recommendations](https://mlsanity.com/paper/2405.19345/) (2024)

## Abstract (arXiv)

Designing learning systems which are invariant to certain data transformations is critical in machine learning. Practitioners can typically enforce a desired invariance on the trained model through the choice of a network architecture, e.g. using convolutions for translations, or using data augmentation. Yet, enforcing true invariance in the network can be difficult, and data invariances are not always known a piori. State-of-the-art methods for learning data augmentation policies require held-out data and are based on bilevel optimization problems, which are complex to solve and often computationally demanding. In this work we investigate new ways of learning invariances only from the training data. Using learnable augmentation layers built directly in the network, we demonstrate that our method is very versatile. It can incorporate any type of differentiable augmentation and be applied to a broad class of learning problems beyond computer vision. We provide empirical evidence showing that our approach is easier and faster to train than modern automatic data augmentation techniques based on bilevel optimization, while achieving comparable results. Experiments show that while the invariances transferred to a model through automatic data augmentation are limited by the model expressivity, the invariance yielded by our approach is insensitive to it by design.