# ERM++: An Improved Baseline for Domain Generalization

arXiv: https://arxiv.org/abs/2304.01973 · HTML: https://mlsanity.com/paper/2304.01973/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 89 references that are in the map.
- First posted on arXiv 4 Apr 2023.

## Builds on (most cited first, 30 of 67)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)

## Built on by (most cited first, 18 of 18)

- [Towards Modality Generalization: A Benchmark and Prospective Analysis](https://mlsanity.com/paper/2412.18277/) (2024)
- [Is Large-Scale Pretraining the Secret to Good Domain Generalization?](https://mlsanity.com/paper/2412.02856/) (2024)
- [Prompt-driven Latent Domain Generalization for Medical Image Classification](https://mlsanity.com/paper/2401.03002/) (2024)
- [Robust Fine-Tuning of Vision-Language Models for Domain Generalization](https://mlsanity.com/paper/2311.02236/) (2023)
- [The Pursuit of Human Labeling: A New Perspective on Unsupervised Learning](https://mlsanity.com/paper/2311.02940/) (2023)
- [Grounding Stylistic Domain Generalization with Quantitative Domain Shift Measures and Synthetic Scene Images](https://mlsanity.com/paper/2405.15961/) (2024)
- [What's in a Latent? Leveraging Diffusion Latent Space for Domain Generalization](https://mlsanity.com/paper/2503.06698/) (2025)
- [Domain Generalization Under Posterior Drift](https://mlsanity.com/paper/2510.04441/) (2025)
- [Diverse Feature Learning by Self-distillation and Reset](https://mlsanity.com/paper/2403.19941/) (2024)
- [Noise-Aware Generalization: Robustness to In-Domain Noise and Out-of-Domain Generalization](https://mlsanity.com/paper/2504.02996/) (2025)
- [Distribution Shift Is Key to Learning Invariant Prediction](https://mlsanity.com/paper/2601.12296/) (2026)
- [PAS: Estimating the target accuracy before domain adaptation](https://mlsanity.com/paper/2604.09863/) (2026)
- [CPCANet: Deep Unfolding Common Principal Component Analysis for Domain Generalization](https://mlsanity.com/paper/2605.05136/) (2026)
- [Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization](https://mlsanity.com/paper/2605.10183/) (2026)
- [Experience with Single Domain Generalization in Real World Medical Imaging Deployments](https://mlsanity.com/paper/2601.16359/) (2026)
- [From Global to Local: Rethinking CLIP Feature Aggregation for Person Re-Identification](https://mlsanity.com/paper/2604.22190/) (2026)
- [CRISP: Compositional Relations as Invariant Structural Priors for Domain Generalization](https://mlsanity.com/paper/2605.06043/) (2026)
- [Domain Generalization for Smartphone-Based Human Activity Recognition: A Systematic Analysis of Components and Interactions](https://mlsanity.com/paper/2609.14863/) (2026)

## Abstract (arXiv)

Domain Generalization (DG) aims to develop classifiers that can generalize to new, unseen data distributions, a critical capability when collecting new domain-specific data is impractical. A common DG baseline minimizes the empirical risk on the source domains. Recent studies have shown that this approach, known as Empirical Risk Minimization (ERM), can outperform most more complex DG methods when properly tuned. However, these studies have primarily focused on a narrow set of hyperparameters, neglecting other factors that can enhance robustness and prevent overfitting and catastrophic forgetting, properties which are critical for strong DG performance. In our investigation of training data utilization (i.e., duration and setting validation splits), initialization, and additional regularizers, we find that tuning these previously overlooked factors significantly improves model generalization across diverse datasets without adding much complexity. We call this improved, yet simple baseline ERM++. Despite its ease of implementation, ERM++ improves DG performance by over 5% compared to prior ERM baselines on a standard benchmark of 5 datasets with a ResNet-50 and over 15% with a ViT-B/16. It also outperforms all state-of-the-art methods on DomainBed datasets with both architectures. Importantly, ERM++ is easy to integrate into existing frameworks like DomainBed, making it a practical and powerful tool for researchers and practitioners. Overall, ERM++ challenges the need for more complex DG methods by providing a stronger, more reliable baseline that maintains simplicity and ease of use. Code is available at urlhttps://github.com/piotr-teterwak/ermplusplus