# In Search of Forgotten Domain Generalization

arXiv: https://arxiv.org/abs/2410.08258 · HTML: https://mlsanity.com/paper/2410.08258/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2024.

## Built on by (most cited first, 21 of 21)

- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [Understanding LoRA as Knowledge Memory: An Empirical Analysis](https://mlsanity.com/paper/2603.01097/) (2026)
- [MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model](https://mlsanity.com/paper/2510.11653/) (2025)
- [Is Large-Scale Pretraining the Secret to Good Domain Generalization?](https://mlsanity.com/paper/2412.02856/) (2024)
- [When and How Does CLIP Enable Domain and Compositional Generalization?](https://mlsanity.com/paper/2502.09507/) (2025)
- [Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research](https://mlsanity.com/paper/2504.13101/) (2025)
- [Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Prediction](https://mlsanity.com/paper/2506.17503/) (2025)
- [Few-Shot, Now for Real: Medical VLMs Adaptation without Balanced Sets or Validation](https://mlsanity.com/paper/2506.17500/) (2025)
- [Category Discovery: An Open-World Perspective](https://mlsanity.com/paper/2509.22542/) (2025)
- [Thousand-Brains Systems: Sensorimotor Intelligence for Rapid, Robust Learning and Inference](https://mlsanity.com/paper/2507.04494/) (2025)
- [LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models](https://mlsanity.com/paper/2506.16950/) (2025)
- [Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training](https://mlsanity.com/paper/2602.10815/) (2026)
- [Modularity is the Bedrock of Natural and Artificial Intelligence](https://mlsanity.com/paper/2602.18960/) (2026)
- [Pruning the Paradox: How CLIP's Most Informative Heads Enhance Performance While Amplifying Bias](https://mlsanity.com/paper/2503.11103/) (2025)
- [Beyond Accuracy: What Matters in Designing Well-Behaved Image Classification Models?](https://mlsanity.com/paper/2503.17110/) (2025)
- [Meta-Learning Transformers to Improve In-Context Generalization](https://mlsanity.com/paper/2507.05019/) (2025)
- [Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representations](https://mlsanity.com/paper/2508.21769/) (2025)
- [Generalized Category Discovery under Domain Shifts: From Vision to Vision-Language Models](https://mlsanity.com/paper/2605.00906/) (2026)
- [Vertical Fusion: Condensing Internal Representations for Robust ViT Classification](https://mlsanity.com/paper/2607.10391/) (2026)
- [Domain-Aware Pruning: Sparsity and Domain Generalization via Regularized Probabilistic Masking](https://mlsanity.com/paper/2608.08624/) (2026)
- [LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure](https://mlsanity.com/paper/2608.13545/) (2026)

## Abstract (arXiv)

Out-of-Domain (OOD) generalization is the ability of a model trained on one or more domains to generalize to unseen domains. In the ImageNet era of computer vision, evaluation sets for measuring a model's OOD performance were designed to be strictly OOD with respect to style. However, the emergence of foundation models and expansive web-scale datasets has obfuscated this evaluation process, as datasets cover a broad range of domains and risk test domain contamination. In search of the forgotten domain generalization, we create large-scale datasets subsampled from LAION -- LAION-Natural and LAION-Rendition -- that are strictly OOD to corresponding ImageNet and DomainNet test sets in terms of style. Training CLIP models on these datasets reveals that a significant portion of their performance is explained by in-domain examples. This indicates that the OOD generalization challenges from the ImageNet era still prevail and that training on web-scale data merely creates the illusion of OOD generalization. Furthermore, through a systematic exploration of combining natural and rendition datasets in varying proportions, we identify optimal mixing ratios for model generalization across these domains. Our datasets and results re-enable meaningful assessment of OOD robustness at scale -- a crucial prerequisite for improving model robustness.