# NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalization

arXiv: https://arxiv.org/abs/2109.02038 · HTML: https://mlsanity.com/paper/2109.02038/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 47 references that are in the map.
- First posted on arXiv 5 Sep 2021.

## Builds on (most cited first, 30 of 41)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Domain Generalization via Model-Agnostic Learning of Semantic Features](https://mlsanity.com/paper/1910.13580/) (2019)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Learning to Generate Novel Domains for Domain Generalization](https://mlsanity.com/paper/2007.03304/) (2020)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection](https://mlsanity.com/paper/1904.07392/) (2019)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [Generalizing Across Domains via Cross-Gradient Training](https://mlsanity.com/paper/1804.10745/) (2018)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [Invariant Risk Minimization Games](https://mlsanity.com/paper/2002.04692/) (2020)
- [Domain Generalization by Marginal Transfer Learning](https://mlsanity.com/paper/1711.07910/) (2017)
- [Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?](https://mlsanity.com/paper/2106.02890/) (2021)

## Built on by (most cited first, 28 of 28)

- [Sparse Mixture-of-Experts are Domain Generalizable Learners](https://mlsanity.com/paper/2206.04046/) (2022)
- [Model Agnostic Sample Reweighting for Out-of-Distribution Learning](https://mlsanity.com/paper/2301.09819/) (2023)
- [Overcoming Simplicity Bias in Deep Networks using a Feature Sieve](https://mlsanity.com/paper/2301.13293/) (2023)
- [Context-Aware Robust Fine-Tuning](https://mlsanity.com/paper/2211.16175/) (2022)
- [DomainDrop: Suppressing Domain-Sensitive Channels for Domain Generalization](https://mlsanity.com/paper/2308.10285/) (2023)
- [On Modality Bias Recognition and Reduction](https://mlsanity.com/paper/2202.12690/) (2022)
- [NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research](https://mlsanity.com/paper/2211.11747/) (2022)
- [G-NAS: Generalizable Neural Architecture Search for Single Domain Generalization Object Detection](https://mlsanity.com/paper/2402.04672/) (2024)
- [Training Debiased Subnetworks with Contrastive Weight Pruning](https://mlsanity.com/paper/2210.05247/) (2022)
- [Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalization](https://mlsanity.com/paper/2306.02595/) (2023)
- [HYPO: Hyperspherical Out-of-Distribution Generalization](https://mlsanity.com/paper/2402.07785/) (2024)
- [ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization](https://mlsanity.com/paper/2210.09236/) (2022)
- [OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images](https://mlsanity.com/paper/2304.10266/) (2023)
- [AutoGPart: Intermediate Supervision Search for Generalizable 3D Part Segmentation](https://mlsanity.com/paper/2203.06558/) (2022)
- [Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes](https://mlsanity.com/paper/2412.13573/) (2024)
- [Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameters](https://mlsanity.com/paper/2310.07361/) (2023)
- [HyperTime: Hyperparameter Optimization for Combating Temporal Distribution Shifts](https://mlsanity.com/paper/2305.18421/) (2023)
- [FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness](https://mlsanity.com/paper/2402.19102/) (2024)
- [Image Background Serves as Good Proxy for Out-of-distribution Data](https://mlsanity.com/paper/2307.00519/) (2023)
- [A Survey of Supernet Optimization and its Applications: Spatial and Temporal Optimization for Neural Architecture Search](https://mlsanity.com/paper/2204.03916/) (2022)
- [Debiasing surgeon: fantastic weights and how to find them](https://mlsanity.com/paper/2403.14200/) (2024)
- [NeuralOOD: Improving Out-of-Distribution Generalization Performance with Brain-machine Fusion Learning Framework](https://mlsanity.com/paper/2408.14950/) (2024)
- [Constrained Maximum Cross-Domain Likelihood for Domain Generalization](https://mlsanity.com/paper/2210.04155/) (2022)
- [Evolutionary Computation for the Design and Enrichment of General-Purpose Artificial Intelligence Systems: Survey and Prospects](https://mlsanity.com/paper/2407.08745/) (2024)
- [Free Lunch for Generating Effective Outlier Supervision](https://mlsanity.com/paper/2301.06657/) (2023)
- [Debiasify: Self-Distillation for Unsupervised Bias Mitigation](https://mlsanity.com/paper/2411.00711/) (2024)
- [Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights](https://mlsanity.com/paper/2501.03782/) (2025)
- [HERCULES: Hardware-Efficient, Robust, Continual Learning Neural Architecture Search](https://mlsanity.com/paper/2605.04103/) (2026)

## Abstract (arXiv)

Recent advances on Out-of-Distribution (OoD) generalization reveal the robustness of deep learning models against distribution shifts. However, existing works focus on OoD algorithms, such as invariant risk minimization, domain generalization, or stable learning, without considering the influence of deep model architectures on OoD generalization, which may lead to sub-optimal performance. Neural Architecture Search (NAS) methods search for architecture based on its performance on the training data, which may result in poor generalization for OoD tasks. In this work, we propose robust Neural Architecture Search for OoD generalization (NAS-OoD), which optimizes the architecture with respect to its performance on generated OoD data by gradient descent. Specifically, a data generator is learned to synthesize OoD data by maximizing losses computed by different neural architectures, while the goal for architecture search is to find the optimal architecture parameters that minimize the synthetic OoD data losses. The data generator and the neural architecture are jointly optimized in an end-to-end manner, and the minimax training process effectively discovers robust architectures that generalize well for different distribution shifts. Extensive experimental results show that NAS-OoD achieves superior performance on various OoD generalization benchmarks with deep models having a much fewer number of parameters. In addition, on a real industry dataset, the proposed NAS-OoD method reduces the error rate by more than 70% compared with the state-of-the-art method, demonstrating the proposed method's practicality for real applications.