# Distributionally Robust Models with Parametric Likelihood Ratios

arXiv: https://arxiv.org/abs/2204.06340 · HTML: https://mlsanity.com/paper/2204.06340/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 53 references that are in the map.
- First posted on arXiv 13 Apr 2022.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Confounding variables can degrade generalization performance of radiological deep learning models](https://mlsanity.com/paper/1807.00431/) (2018)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [Large-Scale Methods for Distributionally Robust Optimization](https://mlsanity.com/paper/2010.05893/) (2020)
- [Simple data balancing achieves competitive worst-group-accuracy](https://mlsanity.com/paper/2110.14503/) (2021)
- [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019)
- [The Mechanics of n-Player Differentiable Games](https://mlsanity.com/paper/1802.05642/) (2018)
- [Training Tips for the Transformer Model](https://mlsanity.com/paper/1804.00247/) (2018)
- [Examining and Combating Spurious Features under Distribution Shift](https://mlsanity.com/paper/2106.07171/) (2021)
- [Distributionally Robust Losses for Latent Covariate Mixtures](https://mlsanity.com/paper/2007.13982/) (2020)
- [MTNT: A Testbed for Machine Translation of Noisy Text](https://mlsanity.com/paper/1809.00388/) (2018)
- [Learning with Average Top-k Loss](https://mlsanity.com/paper/1705.08826/) (2017)
- [Adaptive Sampling for Stochastic Risk-Averse Learning](https://mlsanity.com/paper/1910.12511/) (2019)

## Built on by (most cited first, 17 of 17)

- [Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization](https://mlsanity.com/paper/2407.07880/) (2024)
- [UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup](https://mlsanity.com/paper/2209.08928/) (2022)
- [LongCat-Flash-Omni Technical Report](https://mlsanity.com/paper/2511.00279/) (2025)
- [Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets](https://mlsanity.com/paper/2310.04413/) (2023)
- [Understanding Contrastive Learning via Distributionally Robust Optimization](https://mlsanity.com/paper/2310.11048/) (2023)
- [Learning from Heterogeneity: Generalizing Dynamic Facial Expression Recognition via Distributionally Robust Optimization](https://mlsanity.com/paper/2507.15765/) (2025)
- [Learning from a Biased Sample](https://mlsanity.com/paper/2209.01754/) (2022)
- [Diverse Prototypical Ensembles Improve Robustness to Subpopulation Shift](https://mlsanity.com/paper/2505.23027/) (2025)
- [On the Fairness ROAD: Robust Optimization for Adversarial Debiasing](https://mlsanity.com/paper/2310.18413/) (2023)
- [Decision Making under the Exponential Family: Distributionally Robust Optimisation with Bayesian Ambiguity Sets](https://mlsanity.com/paper/2411.16829/) (2024)
- [Distributionally Robust Optimization via Generative Ambiguity Modeling](https://mlsanity.com/paper/2602.08976/) (2026)
- [Reweighted Mixup for Subpopulation Shift](https://mlsanity.com/paper/2304.04148/) (2023)
- [Modeling the Q-Diversity in a Min-max Play Game for Robust Optimization](https://mlsanity.com/paper/2305.12123/) (2023)
- [Decision Making under Model Misspecification: DRO with Robust Bayesian Ambiguity Sets](https://mlsanity.com/paper/2505.03585/) (2025)
- [Distributionally Robust Optimisation with Bayesian Ambiguity Sets](https://mlsanity.com/paper/2409.03492/) (2024)
- [Imagine Beyond! Distributionally Robust Auto-Encoding for State Space Coverage in Online Reinforcement Learning](https://mlsanity.com/paper/2505.17830/) (2025)
- [Generative Distributionally Robust Optimization](https://mlsanity.com/paper/2607.24983/) (2026)

## Abstract (arXiv)

As machine learning models are deployed ever more broadly, it becomes increasingly important that they are not only able to perform well on their training distribution, but also yield accurate predictions when confronted with distribution shift. The Distributionally Robust Optimization (DRO) framework proposes to address this issue by training models to minimize their expected risk under a collection of distributions, to imitate test-time shifts. This is most commonly achieved by instance-level re-weighting of the training objective to emulate the likelihood ratio with possible test distributions, which allows for estimating their empirical risk via importance sampling (assuming that they are subpopulations of the training distribution). However, re-weighting schemes in the literature are usually limited due to the difficulty of keeping the optimization problem tractable and the complexity of enforcing normalization constraints. In this paper, we show that three simple ideas -- mini-batch level normalization, a KL penalty and simultaneous gradient updates -- allow us to train models with DRO using a broader class of parametric likelihood ratios. In a series of experiments on both image and text classification benchmarks, we find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches, and that the method performs reliably well with little hyper-parameter tuning. Code to reproduce our experiments can be found at https://github.com/pmichel31415/P-DRO.