# Toward Learning Robust and Invariant Representations with Alignment Regularization and Data Augmentation

arXiv: https://arxiv.org/abs/2206.01909 · HTML: https://mlsanity.com/paper/2206.01909/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 73 references that are in the map.
- First posted on arXiv 4 Jun 2022.

## Builds on (most cited first, 30 of 50)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [Self-Challenging Improves Cross-Domain Generalization](https://mlsanity.com/paper/2007.02454/) (2020)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks](https://mlsanity.com/paper/1905.13545/) (2019)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [Adversarial Logit Pairing](https://mlsanity.com/paper/1803.06373/) (2018)
- [Fast Computation of Wasserstein Barycenters](https://mlsanity.com/paper/1310.4375/) (2013)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [How Does Mixup Help With Robustness and Generalization?](https://mlsanity.com/paper/2010.04819/) (2020)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network](https://mlsanity.com/paper/1806.09231/) (2018)

## Built on by (most cited first, 11 of 11)

- [Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shift](https://mlsanity.com/paper/2211.02843/) (2022)
- [Automatic Data Augmentation via Invariance-Constrained Learning](https://mlsanity.com/paper/2209.15031/) (2022)
- [Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models](https://mlsanity.com/paper/2311.01441/) (2023)
- [SIPDO: Closed-Loop Prompt Optimization via Synthetic Data Feedback](https://mlsanity.com/paper/2505.19514/) (2025)
- [Toward Robust Diagnosis: A Contour Attention Preserving Adversarial Defense for COVID-19 Detection](https://mlsanity.com/paper/2211.16806/) (2022)
- [Towards Deep Learning Models Resistant to Transfer-based Adversarial Attacks via Data-centric Robust Learning](https://mlsanity.com/paper/2310.09891/) (2023)
- [Robustar: Interactive Toolbox Supporting Precise Data Annotation for Robust Vision Learning](https://mlsanity.com/paper/2207.08944/) (2022)
- [Incorporating Supervised Domain Generalization into Data Augmentation](https://mlsanity.com/paper/2310.01029/) (2023)
- [Approximate Nullspace Augmented Finetuning for Robust Vision Transformers](https://mlsanity.com/paper/2403.10476/) (2024)
- [Quantitative Evaluation of the Saliency Map for Alzheimer's Disease Classifier with Anatomical Segmentation](https://mlsanity.com/paper/2407.08546/) (2024)
- [A Mathematics Framework of Artificial Shifted Population Risk and Its Further Understanding Related to Consistency Regularization](https://mlsanity.com/paper/2502.10723/) (2025)

## Abstract (arXiv)

Data augmentation has been proven to be an effective technique for developing machine learning models that are robust to known classes of distributional shifts (e.g., rotations of images), and alignment regularization is a technique often used together with data augmentation to further help the model learn representations invariant to the shifts used to augment the data. In this paper, motivated by a proliferation of options of alignment regularizations, we seek to evaluate the performances of several popular design choices along the dimensions of robustness and invariance, for which we introduce a new test procedure. Our synthetic experiment results speak to the benefits of squared l2 norm regularization. Further, we also formally analyze the behavior of alignment regularization to complement our empirical study under assumptions we consider realistic. Finally, we test this simple technique we identify (worst-case data augmentation with squared l2 norm alignment regularization) and show that the benefits of this method outrun those of the specially designed methods. We also release a software package in both TensorFlow and PyTorch for users to use the method with a couple of lines at https://github.com/jyanln/AlignReg.