# Does Distributionally Robust Supervised Learning Give Robust Classifiers?

arXiv: https://arxiv.org/abs/1611.02041 · HTML: https://mlsanity.com/paper/1611.02041/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2016.

## Built on by (most cited first, 30 of 38)

- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Model-Based Domain Generalization](https://mlsanity.com/paper/2102.11436/) (2021)
- [Label-Imbalanced and Group-Sensitive Classification under Overparameterization](https://mlsanity.com/paper/2103.01550/) (2021)
- [Discover and Cure: Concept-aware Mitigation of Spurious Correlation](https://mlsanity.com/paper/2305.00650/) (2023)
- [PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains](https://mlsanity.com/paper/2102.12206/) (2021)
- [Distributionally Robust Learning](https://mlsanity.com/paper/2108.08993/) (2021)
- [An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization](https://mlsanity.com/paper/2102.13128/) (2021)
- [Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives](https://mlsanity.com/paper/2307.16851/) (2023)
- [Focus on the Common Good: Group Distributional Robustness Follows](https://mlsanity.com/paper/2110.02619/) (2021)
- [Kernelized Heterogeneous Risk Minimization](https://mlsanity.com/paper/2110.12425/) (2021)
- [Distributionally Robust Models with Parametric Likelihood Ratios](https://mlsanity.com/paper/2204.06340/) (2022)
- [Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers](https://mlsanity.com/paper/2105.12628/) (2021)
- [Algorithmic Bias and Data Bias: Understanding the Relation between Distributionally Robust Optimization and Data Curation](https://mlsanity.com/paper/2106.09467/) (2021)
- [Model Generalization on COVID-19 Fake News Detection](https://mlsanity.com/paper/2101.03841/) (2021)
- [Robust Fairness-aware Learning Under Sample Selection Bias](https://mlsanity.com/paper/2105.11570/) (2021)
- [Semantically Distributed Robust Optimization for Vision-and-Language Inference](https://mlsanity.com/paper/2110.07165/) (2021)
- [Latent Class-Conditional Noise Model](https://mlsanity.com/paper/2302.09595/) (2023)
- [A Holistic Assessment of the Reliability of Machine Learning Systems](https://mlsanity.com/paper/2307.10586/) (2023)
- [Learning Stable Classifiers by Transferring Unstable Features](https://mlsanity.com/paper/2106.07847/) (2021)
- [Pulling Up by the Causal Bootstraps: Causal Data Augmentation for Pre-training Debiasing](https://mlsanity.com/paper/2108.12510/) (2021)
- [Adversarial Regression with Doubly Non-negative Weighting Matrices](https://mlsanity.com/paper/2109.14875/) (2021)
- [On Certifying and Improving Generalization to Unseen Domains](https://mlsanity.com/paper/2206.12364/) (2022)
- [Stochastic Approximation Approaches to Group Distributionally Robust Optimization and Beyond](https://mlsanity.com/paper/2302.09267/) (2023)
- [Identifying and Benchmarking Natural Out-of-Context Prediction Problems](https://mlsanity.com/paper/2110.13223/) (2021)
- [Distributional Shift Adaptation using Domain-Specific Features](https://mlsanity.com/paper/2211.04670/) (2022)
- [Reweighted Mixup for Subpopulation Shift](https://mlsanity.com/paper/2304.04148/) (2023)
- [Explainability-aided Domain Generalization for Image Classification](https://mlsanity.com/paper/2104.01742/) (2021)
- [Improving Baselines in the Wild](https://mlsanity.com/paper/2112.15550/) (2021)
- [Skeptical binary inferences in multi-label problems with sets of probabilities](https://mlsanity.com/paper/2205.00662/) (2022)
- [Robust Learning in Heterogeneous Contexts](https://mlsanity.com/paper/2105.08532/) (2021)

## Abstract (arXiv)

Distributionally Robust Supervised Learning (DRSL) is necessary for building reliable machine learning systems. When machine learning is deployed in the real world, its performance can be significantly degraded because test data may follow a different distribution from training data. DRSL with f-divergences explicitly considers the worst-case distribution shift by minimizing the adversarially reweighted training loss. In this paper, we analyze this DRSL, focusing on the classification scenario. Since the DRSL is explicitly formulated for a distribution shift scenario, we naturally expect it to give a robust classifier that can aggressively handle shifted distributions. However, surprisingly, we prove that the DRSL just ends up giving a classifier that exactly fits the given training distribution, which is too pessimistic. This pessimism comes from two sources: the particular losses used in classification and the fact that the variety of distributions to which the DRSL tries to be robust is too wide. Motivated by our analysis, we propose simple DRSL that overcomes this pessimism and empirically demonstrate its effectiveness.