# Estimating Generalization under Distribution Shifts via Domain-Invariant Representations

arXiv: https://arxiv.org/abs/2007.03511 · HTML: https://mlsanity.com/paper/2007.03511/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jul 2020.

## Built on by (most cited first, 30 of 56)

- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs](https://mlsanity.com/paper/2202.05441/) (2022)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Fair Mixup: Fairness via Interpolation](https://mlsanity.com/paper/2103.06503/) (2021)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [Exploiting Domain-Specific Features to Enhance Domain Generalization](https://mlsanity.com/paper/2110.09410/) (2021)
- [Size-Invariant Graph Representations for Graph Classification Extrapolations](https://mlsanity.com/paper/2103.05045/) (2021)
- [Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles](https://mlsanity.com/paper/2106.15728/) (2021)
- [Predicting Out-of-Distribution Error with the Projection Norm](https://mlsanity.com/paper/2202.05834/) (2022)
- [Tree Mover's Distance: Bridging Graph Metrics and Stability of Graph Neural Networks](https://mlsanity.com/paper/2210.01906/) (2022)
- [Measuring Generalization with Optimal Transport](https://mlsanity.com/paper/2106.03314/) (2021)
- [Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation](https://mlsanity.com/paper/2302.01094/) (2023)
- [A Survey on Evaluation of Out-of-Distribution Generalization](https://mlsanity.com/paper/2403.01874/) (2024)
- [DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networks](https://mlsanity.com/paper/2302.14685/) (2023)
- [Estimating and Explaining Model Performance When Both Covariates and Labels Shift](https://mlsanity.com/paper/2209.08436/) (2022)
- [Energy-based Automated Model Evaluation](https://mlsanity.com/paper/2401.12689/) (2024)
- [(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancy](https://mlsanity.com/paper/2306.00312/) (2023)
- [Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Features](https://mlsanity.com/paper/2111.03740/) (2021)
- [A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation](https://mlsanity.com/paper/2406.07320/) (2024)
- [Handling Out-of-Distribution Data: A Survey](https://mlsanity.com/paper/2507.21160/) (2025)
- [Three New Validators and a Large-Scale Benchmark Ranking for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2208.07360/) (2022)
- [AETTA: Label-Free Accuracy Estimation for Test-Time Adaptation](https://mlsanity.com/paper/2404.01351/) (2024)
- [Towards Evaluating the Robustness of Neural Networks Learned by Transduction](https://mlsanity.com/paper/2110.14735/) (2021)
- [Moderately Distributional Exploration for Domain Generalization](https://mlsanity.com/paper/2304.13976/) (2023)
- [MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts](https://mlsanity.com/paper/2405.18979/) (2024)
- [ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models](https://mlsanity.com/paper/2307.00398/) (2023)
- [Selecting which Dense Retriever to use for Zero-Shot Search](https://mlsanity.com/paper/2309.09403/) (2023)
- [Predicting the Performance of Foundation Models via Agreement-on-the-Line](https://mlsanity.com/paper/2404.01542/) (2024)
- [HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions](https://mlsanity.com/paper/2209.08443/) (2022)
- [CAME: Contrastive Automated Model Evaluation](https://mlsanity.com/paper/2308.11111/) (2023)

## Abstract (arXiv)

When machine learning models are deployed on a test distribution different from the training distribution, they can perform poorly, but overestimate their performance. In this work, we aim to better estimate a model's performance under distribution shift, without supervision. To do so, we use a set of domain-invariant predictors as a proxy for the unknown, true target labels. Since the error of the resulting risk estimate depends on the target risk of the proxy model, we study generalization of domain-invariant representations and show that the complexity of the latent representation has a significant influence on the target risk. Empirically, our approach (1) enables self-tuning of domain adaptation models, and (2) accurately estimates the target error of given models under distribution shift. Other applications include model selection, deciding early stopping and error detection.