# In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness

arXiv: https://arxiv.org/abs/2012.04550 · HTML: https://mlsanity.com/paper/2012.04550/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2020.

## Built on by (most cited first, 30 of 50)

- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [Surgical Fine-Tuning Improves Adaptation to Distribution Shifts](https://mlsanity.com/paper/2210.11466/) (2022)
- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [Cycle Self-Training for Domain Adaptation](https://mlsanity.com/paper/2103.03571/) (2021)
- [Test-Time Adaptation via Conjugate Pseudo-labels](https://mlsanity.com/paper/2207.09640/) (2022)
- [Extending the WILDS Benchmark for Unsupervised Adaptation](https://mlsanity.com/paper/2112.05090/) (2021)
- [Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?](https://mlsanity.com/paper/2106.02890/) (2021)
- [Break-It-Fix-It: Unsupervised Learning for Program Repair](https://mlsanity.com/paper/2106.06600/) (2021)
- [ZIN: When and How to Learn Invariance Without Environment Partition?](https://mlsanity.com/paper/2203.05818/) (2022)
- [Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection](https://mlsanity.com/paper/2306.09158/) (2023)
- [Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations](https://mlsanity.com/paper/2204.02683/) (2022)
- [ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets](https://mlsanity.com/paper/2209.00613/) (2022)
- [Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalization](https://mlsanity.com/paper/2206.07766/) (2022)
- [Trainable Projected Gradient Method for Robust Fine-tuning](https://mlsanity.com/paper/2303.10720/) (2023)
- [If your data distribution shifts, use self-learning](https://mlsanity.com/paper/2104.12928/) (2021)
- [Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift](https://mlsanity.com/paper/2207.08977/) (2022)
- [Identifying Weight-Variant Latent Causal Models](https://mlsanity.com/paper/2208.14153/) (2022)
- [Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?](https://mlsanity.com/paper/2401.13544/) (2024)
- [SPA: A Graph Spectral Alignment Perspective for Domain Adaptation](https://mlsanity.com/paper/2310.17594/) (2023)
- [Self-training Converts Weak Learners to Strong Learners in Mixture Models](https://mlsanity.com/paper/2106.13805/) (2021)
- [Out-of-Distribution Generalization in Time Series: A Survey](https://mlsanity.com/paper/2503.13868/) (2025)
- [Pareto Navigation Gradient Descent: a First-Order Algorithm for Optimization in Pareto Set](https://mlsanity.com/paper/2110.08713/) (2021)
- [VIBE: Topic-Driven Temporal Adaptation for Twitter Classification](https://mlsanity.com/paper/2310.10191/) (2023)
- [Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback](https://mlsanity.com/paper/2309.15762/) (2023)
- [Distribution Shifts at Scale: Out-of-distribution Detection in Earth Observation](https://mlsanity.com/paper/2412.13394/) (2024)
- [Exploiting Position Bias for Robust Aspect Sentiment Classification](https://mlsanity.com/paper/2105.14210/) (2021)
- [Towards Understanding GD with Hard and Conjugate Pseudo-labels for Test-Time Adaptation](https://mlsanity.com/paper/2210.10019/) (2022)
- [General Greedy De-bias Learning](https://mlsanity.com/paper/2112.10572/) (2021)
- [Breaking Correlation Shift via Conditional Invariant Regularizer](https://mlsanity.com/paper/2207.06687/) (2022)

## Abstract (arXiv)

Consider a prediction setting with few in-distribution labeled examples and many unlabeled examples both in- and out-of-distribution (OOD). The goal is to learn a model which performs well both in-distribution and OOD. In these settings, auxiliary information is often cheaply available for every input. How should we best leverage this auxiliary information for the prediction task? Empirically across three image and time-series datasets, and theoretically in a multi-task linear regression setting, we show that (i) using auxiliary information as input features improves in-distribution error but can hurt OOD error; but (ii) using auxiliary information as outputs of auxiliary pre-training tasks improves OOD error. To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training). We show both theoretically and empirically that In-N-Out outperforms auxiliary inputs or outputs alone on both in-distribution and OOD error.