mlsanityData to 14 Oct 2026

Learning Optimal Features via Partial Invariance

arXiv 2301.12067 ↗First posted 28 Jan 2023cs.LG cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Learning models that are robust to distribution shifts is a key concern in the context of their real-life applicability. Invariant Risk Minimization (IRM) is a popular framework that aims to learn robust models from multiple environments. The success of IRM requires an important assumption: the underlying causal mechanisms/features remain invariant across environments. When not satisfied, we show that IRM can over-constrain the predictor and to remedy this, we propose a relaxation via textitpartial invariance. In this work, we theoretically highlight the sub-optimality of IRM and then demonstrate how learning from a partition of training domains can help improve invariant models. Several experiments, conducted both in linear settings as well as with deep neural networks on tasks over both language and image data, allow us to verify our conclusions.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 34 of its 50 references that are in the map. First posted on arXiv 28 Jan 2023.

Builds on

34 of 50 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019Invariant Risk Minimizationcited by 1,854
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  7. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  8. 2020In Search of Lost Domain Generalizationcited by 1,054
  9. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  10. 2018Recognition in Terra Incognitacited by 658
  11. 2015Causal inference using invariant prediction: identification and confidence intervalscited by 536
  12. 2019Agnostic Federated Learningcited by 456
22 more
  1. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  2. 2020The Risks of Invariant Risk Minimizationcited by 262
  3. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 253
  4. 2020Domain Generalization using Causal Matchingcited by 225
  5. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  6. 2018Deep Learning: A Critical Appraisalcited by 217
  7. 2020Invariant Risk Minimization Gamescited by 188
  8. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
  9. 2018Domain Generalization via Conditional Invariant Representationcited by 146
  10. 2017Invariant Causal Prediction for Nonlinear Modelscited by 140
  11. 2021Does Invariant Risk Minimization Capture Invariance?cited by 116
  12. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
  13. 2022MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflictscited by 81
  14. 2020Empirical or Invariant Risk Minimization? A Sample Complexity Perspectivecited by 62
  15. 2020Fairness and Robustness in Invariant Learning: A Case Study in Toxicity Classificationcited by 32
  16. 2020Enforcing Predictive Invariance across Structured Biomedical Domainscited by 23
  17. 2021On the benefits of representation regularization in invariance based domain generalizationcited by 17
  18. 2021Invariant Language Modelingcited by 15
  19. 2020Accounting for Unobserved Confounding in Domain Generalizationcited by 14
  20. 2021Leveraging cross-platform data to improve automated hate speech detectioncited by 4
  21. 2022Regularization Penalty Optimization for Addressing Data Quality Variance in OoD Algorithmscited by 4
  22. 2020Quantization Games on Social Networks and Language Evolutioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 3.

5 papers.

  1. 2026Distribution Shift Is Key to Learning Invariant Predictioncited by 2
  2. 2023Efficient Model-Agnostic Multi-Group Equivariant Networkscited by 1
  3. 2025Multi-Disease-Aware Training Strategy for Cardiac MR Image Segmentationcited by 0
  4. 2026How Many Samples Are Enough for Learning Across Domains?cited by 0
  5. 2026Can Domain Generalization be Guaranteed in Small-Sample Learning?cited by 0