mlsanityData to 7 Oct 2026

Optimal Representations for Covariate Shift

arXiv 2201.00057 ↗First posted 31 Dec 2021cs.LG cs.AI cs.ITcited by 51

Not on the map. Why?

Abstract (arXiv)

Machine learning systems often experience a distribution shift between training and testing. In this paper, we introduce a simple variational objective whose optima are exactly the set of all representations on which risk minimizers are guaranteed to be robust to any distribution shift that preserves the Bayes predictor, e.g., covariate shifts. Our objective has two components. First, a representation must remain discriminative for the task, i.e., some predictor must be able to simultaneously minimize the source and target risk. Second, the representation's marginal support needs to be the same across source and target. We make this practical by designing self-supervised objectives that only use unlabelled data and augmentations to train robust representations. Our objectives give insights into the robustness of CLIP, and further improve CLIP's representations to achieve SOTA results on DomainBed.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 45 of its 73 references that are in the map. First posted on arXiv 31 Dec 2021.

Builds on

45 of 73 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  10. 2020Supervised Contrastive Learningcited by 2,803
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2019Invariant Risk Minimizationcited by 1,832
33 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2019Natural Adversarial Examplescited by 1,401
  4. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  5. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  6. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  7. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,194
  8. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  9. 2020In Search of Lost Domain Generalizationcited by 1,042
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  11. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  12. 2016Deep Variational Information Bottleneckcited by 915
  13. 2018Variational image compression with a scale hyperpriorcited by 910
  14. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 737
  15. 2016End-to-end Optimized Image Compressioncited by 673
  16. 2018Recognition in Terra Incognitacited by 655
  17. 2016Deep Transfer Learning with Joint Adaptation Networkscited by 609
  18. 2019On Variational Bounds of Mutual Informationcited by 550
  19. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  20. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  21. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  22. 2017Lossy Image Compression with Compressive Autoencoderscited by 289
  23. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  24. 2020Improve Unsupervised Domain Adaptation with Mixup Trainingcited by 185
  25. 2020Domain Adaptation with Conditional Distribution Matching and Generalized Label Shiftcited by 124
  26. 2019Support and Invertibility in Domain-Invariant Representationscited by 87
  27. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  28. 2019Do Image Classifiers Generalize Across Time?cited by 59
  29. 2021Lossy Compression for Lossless Predictioncited by 51
  30. 2020Minimum Excess Risk in Bayesian Learningcited by 35
  31. 2021KL Guided Domain Adaptationcited by 35
  32. 2020Learning Optimal Representations with the Decodable Information Bottleneckcited by 32
  33. 2021Multimodal Self-Supervised Learning of General Audio Representationscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 18; 2024: 9; 2025: 2; 2026: 5.

51 papers.

Sort:

By area:

  1. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  2. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  3. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  4. 2022NICO++: Towards Better Benchmarking for Domain Generalizationcited by 86
  5. 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
  6. 2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
  7. 2023Improved Test-Time Adaptation for Domain Generalizationcited by 45
  8. 2022Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learningcited by 39
  9. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  10. 2022Improving Self-Supervised Learning by Characterizing Idealized Representationscited by 33
  11. 2022Grounding Visual Representations with Texts for Domain Generalizationcited by 30
  12. 2022WOODS: Benchmarks for Out-of-Distribution Generalization in Time Seriescited by 29
38 more
  1. 2023Distribution Shift Inversion for Out-of-Distribution Predictioncited by 23
  2. 2023MADG: Margin-based Adversarial Learning for Domain Generalizationcited by 22
  3. 2022Weighted Ensemble Self-Supervised Learningcited by 19
  4. 2023Domain Generalization via Rationale Invariancecited by 16
  5. 2023Improving Domain Generalization with Domain Relationscited by 14
  6. 2022Domain Adaptation meets Individual Fairness. And they get alongcited by 13
  7. 2022Back-to-Bones: Rediscovering the Role of Backbones in Domain Generalizationcited by 12
  8. 2022Understanding new tasks through the lens of training data via exponential tiltingcited by 11
  9. 20223DOS: Towards 3D Open Set Learning -- Benchmarking and Understanding Semantic Novelty Detection on Point Cloudscited by 11
  10. 2024Towards Modality Generalization: A Benchmark and Prospective Analysiscited by 10
  11. 2023EPVT: Environment-aware Prompt Vision Transformer for Domain Generalization in Skin Lesion Recognitioncited by 9
  12. 2023Graph Out-of-Distribution Generalization with Controllable Data Augmentationcited by 9
  13. 2024Prompt-driven Latent Domain Generalization for Medical Image Classificationcited by 8
  14. 2024LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalizationcited by 8
  15. 2022A picture of the space of typical learnable taskscited by 6
  16. 2023Evaluating Self-Supervised Learning via Risk Decompositioncited by 6
  17. 2024DPStyler: Dynamic PromptStyler for Source-Free Domain Generalizationcited by 6
  18. 2024PromptTA: Prompt-driven Text Adapter for Source-free Domain Generalizationcited by 6
  19. 2022OOD-Probe: A Neural Interpretation of Out-of-Domain Generalizationcited by 5
  20. 2023An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibrationcited by 5
  21. 2023Predicting Class Distribution Shift for Reliable Domain Adaptive Object Detectioncited by 4
  22. 2023Quantitatively Measuring and Contrastively Exploring Heterogeneity for Domain Generalizationcited by 4
  23. 2023Double Debiased Covariate Shift Adaptation Robust to Density-Ratio Estimationcited by 3
  24. 2024Towards Understanding Variants of Invariant Risk Minimization through the Lens of Calibrationcited by 3
  25. 2022SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Datacited by 2
  26. 2023Robustness May be More Brittle than We Think under Different Degrees of Distribution Shiftscited by 2
  27. 2023Out of the Ordinary: Spectrally Adapting Regression for Covariate Shiftcited by 2
  28. 2024Weighted Risk Invariance: Domain Generalization under Invariant Feature Shiftcited by 2
  29. 2024UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Modelscited by 2
  30. 2025Why Domain Generalization Fail? A View of Necessity and Sufficiencycited by 2
  31. 2023Bayesian Domain Invariant Learning via Posterior Generalization of Parameter Distributionscited by 1
  32. 2024Harnessing the Power of Vicinity-Informed Analysis for Classification under Covariate Shiftcited by 1
  33. 2025Text-Driven Causal Representation Learning for Source-Free Domain Generalizationcited by 1
  34. 2026Generalizable Multimodal Large Language Model Editing via Invariant Trajectory Learningcited by 1
  35. 2026Self-Supervised Learning from Structural Invariancecited by 1
  36. 2022Generalization Beyond Feature Alignment: Concept Activation-Guided Contrastive Learningcited by 0
  37. 2026When Prompts Interact: Assessing Prompt Arithmetic for Deconfounding under Distribution Shiftcited by 0
  38. 2026DeconDTN-Toolkit: A Library for Evaluation and Enhancement of Robustness to Provenance Shiftcited by 0

50 most cited of 51 All 51 (JSON)