# On the Strong Correlation Between Model Invariance and Generalization

arXiv: https://arxiv.org/abs/2207.07065 · HTML: https://mlsanity.com/paper/2207.07065/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 82 of its 117 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 30 of 82)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)

## Built on by (most cited first, 23 of 23)

- [ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations](https://mlsanity.com/paper/2211.01866/) (2022)
- [A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)](https://mlsanity.com/paper/2402.07410/) (2024)
- [Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation](https://mlsanity.com/paper/2302.01094/) (2023)
- [GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labels](https://mlsanity.com/paper/2310.14586/) (2023)
- [Energy-based Automated Model Evaluation](https://mlsanity.com/paper/2401.12689/) (2024)
- [Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget](https://mlsanity.com/paper/2304.10520/) (2023)
- [Unsupervised Discovery of Object-Centric Neural Fields](https://mlsanity.com/paper/2402.07376/) (2024)
- [Online GNN Evaluation Under Test-time Graph Distribution Shifts](https://mlsanity.com/paper/2403.09953/) (2024)
- [LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies](https://mlsanity.com/paper/2407.16067/) (2024)
- [On the Out of Distribution Robustness of Foundation Models in Medical Image Segmentation](https://mlsanity.com/paper/2311.11096/) (2023)
- [Decoupling Positional and Symbolic Attention Behavior in Transformers](https://mlsanity.com/paper/2511.11579/) (2025)
- [Optimizing Calibration by Gaining Aware of Prediction Correctness](https://mlsanity.com/paper/2404.13016/) (2024)
- [Temporal Generalization Estimation in Evolving Graphs](https://mlsanity.com/paper/2404.04969/) (2024)
- [Optimizing Data Augmentation through Bayesian Model Selection](https://mlsanity.com/paper/2505.21813/) (2025)
- [Self-Supervised Learning from Structural Invariance](https://mlsanity.com/paper/2602.02381/) (2026)
- [SEIS: Subspace-based Equivariance and Invariance Scores for Neural Representations](https://mlsanity.com/paper/2602.04054/) (2026)
- [Fine-Grained ImageNet Classification in the Wild](https://mlsanity.com/paper/2303.02400/) (2023)
- [Adaptive Contextual Perception: How to Generalize to New Backgrounds and Ambiguous Objects](https://mlsanity.com/paper/2306.05963/) (2023)
- [What is Intelligence? A Cycle Closure Perspective](https://mlsanity.com/paper/2404.05484/) (2024)
- [FINED: Feed Instance-Wise Information Need with Essential and Disentangled Parametric Knowledge from the Past](https://mlsanity.com/paper/2406.00012/) (2024)
- [Assessing Model Generalization in Vicinity](https://mlsanity.com/paper/2406.09257/) (2024)
- [Using Synthetic Data to estimate the True Error is theoretically and practically doable](https://mlsanity.com/paper/2511.00964/) (2025)
- [Constraint Breeds Generalization: Temporal Dynamics as an Inductive Bias](https://mlsanity.com/paper/2512.23916/) (2025)

## Abstract (arXiv)

Generalization and invariance are two essential properties of any machine learning model. Generalization captures a model's ability to classify unseen data while invariance measures consistency of model predictions on transformations of the data. Existing research suggests a positive relationship: a model generalizing well should be invariant to certain visual factors. Building on this qualitative implication we make two contributions. First, we introduce effective invariance (EI), a simple and reasonable measure of model invariance which does not rely on image labels. Given predictions on a test image and its transformed version, EI measures how well the predictions agree and with what level of confidence. Second, using invariance scores computed by EI, we perform large-scale quantitative correlation studies between generalization and invariance, focusing on rotation and grayscale transformations. From a model-centric view, we observe generalization and invariance of different models exhibit a strong linear relationship, on both in-distribution and out-of-distribution datasets. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets. Apart from these major findings, other minor but interesting insights are also discussed.