mlsanityData to 7 Oct 2026

Toward Learning Robust and Invariant Representations with Alignment Regularization and Data Augmentation

arXiv 2206.01909 ↗First posted 4 Jun 2022cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Data augmentation has been proven to be an effective technique for developing machine learning models that are robust to known classes of distributional shifts (e.g., rotations of images), and alignment regularization is a technique often used together with data augmentation to further help the model learn representations invariant to the shifts used to augment the data. In this paper, motivated by a proliferation of options of alignment regularizations, we seek to evaluate the performances of several popular design choices along the dimensions of robustness and invariance, for which we introduce a new test procedure. Our synthetic experiment results speak to the benefits of squared l2 norm regularization. Further, we also formally analyze the behavior of alignment regularization to complement our empirical study under assumptions we consider realistic. Finally, we test this simple technique we identify (worst-case data augmentation with squared l2 norm alignment regularization) and show that the benefits of this method outrun those of the specially designed methods. We also release a software package in both TensorFlow and PyTorch for users to use the method with a couple of lines at https://github.com/jyanln/AlignReg.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 50 of its 73 references that are in the map. First posted on arXiv 4 Jun 2022.

Builds on

50 of 73 references.

References · 50

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2017Improved Training of Wasserstein GANscited by 2,223
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  8. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  9. 2015Spatial Transformer Networkscited by 1,423
  10. 2017Random Erasing Data Augmentationcited by 1,235
  11. 2016Group Equivariant Convolutional Networkscited by 1,154
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
38 more
  1. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  2. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  3. 2018Robustness May Be at Odds with Accuracycited by 873
  4. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  5. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  6. 2018Albumentations: fast and flexible image augmentationscited by 558
  7. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  8. 2020Self-Challenging Improves Cross-Domain Generalizationcited by 426
  9. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  10. 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
  11. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  12. 2019Fast AutoAugmentcited by 243
  13. 2018Adversarial Logit Pairingcited by 224
  14. 2013Fast Computation of Wasserstein Barycenterscited by 216
  15. 2019Learning De-biased Representations with Biased Representationscited by 199
  16. 2020How Does Mixup Help With Robustness and Generalization?cited by 193
  17. 2019RUBi: Reducing Unimodal Biases in Visual Question Answeringcited by 183
  18. 2018Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Networkcited by 168
  19. 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
  20. 2019Population Based Augmentation: Efficient Learning of Augmentation Policy Schedulescited by 153
  21. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 131
  22. 2019Learning Robust Representations by Projecting Superficial Statistics Outcited by 123
  23. 2018A Kernel Theory of Modern Data Augmentationcited by 106
  24. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  25. 2019Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentationcited by 105
  26. 2020Syntactic Data Augmentation Increases Robustness to Inference Heuristicscited by 85
  27. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 77
  28. 2018Data augmentation instead of explicit regularizationcited by 62
  29. 2018Regularizing Neural Machine Translation by Target-bidirectional Agreementcited by 40
  30. 2018Theoretical Analysis of Adversarial Learning: A Minimax Approachcited by 31
  31. 2017On the Origin of Deep Learningcited by 23
  32. 2019Does Data Augmentation Lead to Positive Margin?cited by 18
  33. 2019Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustnesscited by 18
  34. 2021On Data-Augmentation and Consistency-Based Semi-Supervised Learningcited by 13
  35. 2018Training Augmentation with Adversarial Examples for Robust Speech Recognitioncited by 11
  36. 2018Learning Noise-Invariant Representations for Robust Speech Recognitioncited by 10
  37. 2019Learning robust visual representations using data augmentation invariancecited by 5
  38. 2020Data augmentation and image understandingcited by 1

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 3; 2024: 2; 2025: 2.

11 papers.

  1. 2022Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shiftcited by 38
  2. 2022Automatic Data Augmentation via Invariance-Constrained Learningcited by 17
  3. 2023Distilling Out-of-Distribution Robustness from Vision-Language Foundation Modelscited by 9
  4. 2025SIPDO: Closed-Loop Prompt Optimization via Synthetic Data Feedbackcited by 7
  5. 2022Toward Robust Diagnosis: A Contour Attention Preserving Adversarial Defense for COVID-19 Detectioncited by 3
  6. 2023Towards Deep Learning Models Resistant to Transfer-based Adversarial Attacks via Data-centric Robust Learningcited by 2
  7. 2022Robustar: Interactive Toolbox Supporting Precise Data Annotation for Robust Vision Learningcited by 0
  8. 2023Incorporating Supervised Domain Generalization into Data Augmentationcited by 0
  9. 2024Approximate Nullspace Augmented Finetuning for Robust Vision Transformerscited by 0
  10. 2024Quantitative Evaluation of the Saliency Map for Alzheimer's Disease Classifier with Anatomical Segmentationcited by 0
  11. 2025A Mathematics Framework of Artificial Shifted Population Risk and Its Further Understanding Related to Consistency Regularizationcited by 0