DivIL: Unveiling and Addressing Over-Invariance for Out-of- Distribution Generalization
Not on the map. Why?
Abstract (arXiv)
Out-of-distribution generalization is a common problem that expects the model to perform well in the different distributions even far from the train data. A popular approach to addressing this issue is invariant learning (IL), in which the model is compiled to focus on invariant features instead of spurious features by adding strong constraints during training. However, there are some potential pitfalls of strong invariant constraints. Due to the limited number of diverse environments and over-regularization in the feature space, it may lead to a loss of important details in the invariant features while alleviating the spurious correlations, namely the over-invariance, which can also degrade the generalization performance. We theoretically define the over-invariance and observe that this issue occurs in various classic IL methods. To alleviate this issue, we propose a simple approach Diverse Invariant Learning (DivIL) by adding the unsupervised contrastive learning and the random masking mechanism compensatory for the invariant constraints, which can be applied to various IL methods. Furthermore, we conduct experiments across multiple modalities across 12 datasets and 6 classic models, verifying our over-invariance insight and the effectiveness of our DivIL framework. Our code is available at https://github.com/kokolerk/DivIL.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 56 of its 69 references that are in the map. First posted on arXiv 18 Feb 2025.
Builds on
56 of 69 references.
References · 56
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2018How Powerful are Graph Neural Networks?cited by 4,355
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
- 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
38 more
- 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
- 2015A large annotated corpus for learning natural language inferencecited by 1,837
- 2019Invariant Risk Minimizationcited by 1,832
- 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
- 2020Graph Contrastive Learning with Augmentationscited by 1,203
- 2020In Search of Lost Domain Generalizationcited by 1,042
- 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
- 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
- 2018Disentangling by Factorisingcited by 614
- 2015Causal inference using invariant prediction: identification and confidence intervalscited by 533
- 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
- 2020Explainability in Graph Neural Networks: A Taxonomic Surveycited by 383
- 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
- 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
- 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
- 2020Environment Inference for Invariant Learningcited by 306
- 2020The Risks of Invariant Risk Minimizationcited by 262
- 2020Domain Generalization using Causal Matchingcited by 224
- 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
- 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
- 2022Discovering Invariant Rationales for Graph Neural Networkscited by 186
- 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
- 2019Understanding Attention and Generalization in Graph Neural Networkscited by 169
- 2022Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphscited by 144
- 2022GOOD: A Graph Out-of-Distribution Benchmarkcited by 140
- 2022Data Augmentation for Deep Graph Learning: A Surveycited by 132
- 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
- 2021Does Invariant Risk Minimization Capture Invariance?cited by 116
- 2024Large Language Models Meet NLP: A Surveycited by 102
- 2021Heterogeneous Risk Minimizationcited by 90
- 2017Invariant Causal Prediction for Sequential Datacited by 63
- 2022ZIN: When and How to Learn Invariance Without Environment Partition?cited by 58
- 2020Distributionally Robust Losses for Latent Covariate Mixturescited by 57
- 2022DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotationscited by 54
- 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
- 2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 43
- 2021Towards Unsupervised Domain Generalizationcited by 40
- 2024A Sober Look at the Robustness of CLIPs to Spurious Featurescited by 38
50 most cited of 56 All 56 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.