# Model-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution Data

arXiv: https://arxiv.org/abs/2005.10247 · HTML: https://mlsanity.com/paper/2005.10247/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2020.

## Built on by (most cited first, 27 of 27)

- [Graph Contrastive Learning Automated](https://mlsanity.com/paper/2106.07594/) (2021)
- [Model-Based Domain Generalization](https://mlsanity.com/paper/2102.11436/) (2021)
- [Towards Principled Disentanglement for Domain Generalization](https://mlsanity.com/paper/2111.13839/) (2021)
- [On the effectiveness of adversarial training against common corruptions](https://mlsanity.com/paper/2103.02325/) (2021)
- [Probable Domain Generalization via Quantile Risk Minimization](https://mlsanity.com/paper/2207.09944/) (2022)
- [SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization](https://mlsanity.com/paper/2106.02266/) (2021)
- [Adversarial Robustness with Semi-Infinite Constrained Learning](https://mlsanity.com/paper/2110.15767/) (2021)
- [Probabilistically Robust Learning: Balancing Average- and Worst-case Performance](https://mlsanity.com/paper/2202.01136/) (2022)
- [GSmooth: Certified Robustness against Semantic Transformations via Generalized Randomized Smoothing](https://mlsanity.com/paper/2206.04310/) (2022)
- [Embracing the Disharmony in Medical Imaging: A Simple and Effective Framework for Domain Adaptation](https://mlsanity.com/paper/2103.12857/) (2021)
- [Distributionally Robust Policy Learning via Adversarial Environment Generation](https://mlsanity.com/paper/2107.06353/) (2021)
- [Toward Certified Robustness Against Real-World Distribution Shifts](https://mlsanity.com/paper/2206.03669/) (2022)
- [Learning Augmentation Distributions using Transformed Risk Minimization](https://mlsanity.com/paper/2111.08190/) (2021)
- [Do Deep Networks Transfer Invariances Across Classes?](https://mlsanity.com/paper/2203.09739/) (2022)
- [Robust Graph Neural Networks via Probabilistic Lipschitz Constraints](https://mlsanity.com/paper/2112.07575/) (2021)
- [Attacking Motion Estimation with Adversarial Snow](https://mlsanity.com/paper/2210.11242/) (2022)
- [Semantic Perturbations with Normalizing Flows for Improved Generalization](https://mlsanity.com/paper/2108.07958/) (2021)
- [Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Data](https://mlsanity.com/paper/2405.05468/) (2024)
- [Concept-based Adversarial Attacks: Tricking Humans and Classifiers Alike](https://mlsanity.com/paper/2203.10166/) (2022)
- [Segmentation Consistency Training: Out-of-Distribution Generalization for Medical Image Segmentation](https://mlsanity.com/paper/2205.15428/) (2022)
- [Understanding Overfitting in Adversarial Training via Kernel Regression](https://mlsanity.com/paper/2304.06326/) (2023)
- [Multimodal Representation Alignment for Cross-modal Information Retrieval](https://mlsanity.com/paper/2506.08774/) (2025)
- [Out-of-Distribution Robustness in Deep Learning Compression](https://mlsanity.com/paper/2110.07007/) (2021)
- [Probabilistic Inverse Modeling: An Application in Hydrology](https://mlsanity.com/paper/2210.06213/) (2022)
- [EM for Mixture of Linear Regression with Clustered Data](https://mlsanity.com/paper/2308.11518/) (2023)
- [Built-in Elastic Transformations for Improved Robustness](https://mlsanity.com/paper/2107.09391/) (2021)
- [Wiggling Weights to Improve the Robustness of Classifiers](https://mlsanity.com/paper/2111.09779/) (2021)

## Abstract (arXiv)

While deep learning has resulted in major breakthroughs in many application domains, the frameworks commonly used in deep learning remain fragile to artificially-crafted and imperceptible changes in the data. In response to this fragility, adversarial training has emerged as a principled approach for enhancing the robustness of deep learning with respect to norm-bounded perturbations. However, there are other sources of fragility for deep learning that are arguably more common and less thoroughly studied. Indeed, natural variation such as lighting or weather conditions can significantly degrade the accuracy of trained neural networks, proving that such natural variation presents a significant challenge for deep learning. In this paper, we propose a paradigm shift from perturbation-based adversarial robustness toward model-based robust deep learning. Our objective is to provide general training algorithms that can be used to train deep neural networks to be robust against natural variation in data. Critical to our paradigm is first obtaining a model of natural variation which can be used to vary data over a range of natural conditions. Such models may be either known a priori or else learned from data. In the latter case, we show that deep generative models can be used to learn models of natural variation that are consistent with realistic conditions. We then exploit such models in three novel model-based robust training algorithms in order to enhance the robustness of deep learning with respect to the given model. Our extensive experiments show that across a variety of naturally-occurring conditions and across various datasets, deep neural networks trained with our model-based algorithms significantly outperform both standard deep learning algorithms as well as norm-bounded robust deep learning algorithms.