mlsanityData to 14 Oct 2026

DORO: Distributional and Outlier Robust Optimization

arXiv 2106.06142 ↗First posted 11 Jun 2021cs.LG stat.MLcited by 56

Not on the map. Why?

Abstract (arXiv)

Many machine learning tasks involve subpopulation shift where the testing data distribution is a subpopulation of the training distribution. For such settings, a line of recent work has proposed the use of a variant of empirical risk minimization(ERM) known as distributionally robust optimization (DRO). In this work, we apply DRO to real, large-scale tasks with subpopulation shift, and observe that DRO performs relatively poorly, and moreover has severe instability. We identify one direct cause of this phenomenon: sensitivity of DRO to outliers in the datasets. To resolve this issue, we propose the framework of DORO, for Distributional and Outlier Robust Optimization. At the core of this approach is a refined risk function which prevents DRO from overfitting to potential outliers. We instantiate DORO for the Cressie-Read family of R'enyi divergence, and delve into two specific instances of this family: CVaR and chi2-DRO. We theoretically prove the effectiveness of the proposed method, and empirically show that DORO improves the performance and stability of DRO with experiments on large modern datasets, thereby positively addressing the open question raised by Hashimoto et al., 2018.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 26 of its 49 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

26 of 49 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  4. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  5. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  6. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  7. 2020In Search of Lost Domain Generalizationcited by 1,054
  8. 2017Counterfactual Fairnesscited by 804
  9. 2018Deep Visual Domain Adaptation: A Surveycited by 635
  10. 2018A Survey on Deep Transfer Learningcited by 419
  11. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
  12. 2016Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatmentcited by 369
14 more
  1. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 364
  2. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  3. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  4. 2020Fairness without Demographics through Adversarially Reweighted Learningcited by 251
  5. 2016Robust Estimators in High Dimensions without the Computational Intractabilitycited by 209
  6. 2019Distributionally Robust Language Modelingcited by 143
  7. 2016Agnostic Estimation of Mean and Covariancecited by 131
  8. 2020Deep Visual Domain Adaptationcited by 91
  9. 2017Being Robust (in High Dimensions) Can Be Practicalcited by 83
  10. 2018Robust Estimation via Robust Gradient Estimationcited by 76
  11. 2019Generalized Resilience and Robust Statisticscited by 37
  12. 2020Learning Bounds for Risk-sensitive Learningcited by 33
  13. 2020Class-Weighted Classification: Trade-offs and Robust Approachescited by 30
  14. 2021Modeling the Second Player in Distributionally Robust Optimizationcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 14; 2023: 16; 2024: 12; 2025: 7; 2026: 4.

56 papers.

Sort:

By area:

  1. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 66
  2. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 53
  3. 2023Improving Fairness in Deepfake Detectioncited by 40
  4. 2022UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixupcited by 34
  5. 2022Improving Task-free Continual Learning by Distributionally Robust Memory Evolutioncited by 31
  6. 2022Understanding Why Generalized Reweighting Does Not Improve Over ERMcited by 29
  7. 2023Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimizationcited by 29
  8. 2022Rank-based Decomposable Losses in Machine Learning: A Surveycited by 26
  9. 2021Focus on the Common Good: Group Distributional Robustness Followscited by 25
  10. 2023Understanding Contrastive Learning via Distributionally Robust Optimizationcited by 21
  11. 2023On the Theories Behind Hard Negative Sampling for Recommendationcited by 18
  12. 2022Subgroup Robustness Grows On Trees: An Empirical Baseline Investigationcited by 17
38 more
  1. 2023Distributionally Robust Post-hoc Classifiers under Prior Shiftscited by 16
  2. 2023Outlier-Robust Wasserstein DROcited by 16
  3. 2022Distributionally Robust Optimization via Ball Oracle Accelerationcited by 14
  4. 2022Domain Adaptation meets Individual Fairness. And they get alongcited by 13
  5. 2021Boosted CVaR Classificationcited by 12
  6. 2024Robust COVID-19 Detection in CT Images with CLIPcited by 12
  7. 2022Understanding new tasks through the lens of training data via exponential tiltingcited by 11
  8. 2023Outlier Robust Adversarial Trainingcited by 10
  9. 2024PSL: Rethinking and Improving Softmax Loss from Pairwise Perspective for Recommendationcited by 9
  10. 2023Understanding the robustness difference between stochastic gradient descent and adaptive gradient methodscited by 8
  11. 2023Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimizationcited by 6
  12. 2022Learning with Noisy Labels over Imbalanced Subpopulationscited by 5
  13. 2023Geometry-Calibrated DRO: Combating Over-Pessimism with Free Energy Implicationscited by 5
  14. 2024Improving Generalization via Meta-Learning on Hard Samplescited by 5
  15. 2024Robust Distribution Learning with Local and Global Adversarial Corruptionscited by 5
  16. 2021A Survey of Learning Criteria Going Beyond the Usual Riskcited by 4
  17. 2023Task-Distributionally Robust Data-Free Meta-Learningcited by 4
  18. 2022Flexible risk design using bi-directional dispersioncited by 3
  19. 2022DRFLM: Distributionally Robust Federated Learning with Inter-client Noise via Local Mixupcited by 3
  20. 2022Outlier-Robust Group Inference via Gradient Space Clusteringcited by 3
  21. 2022How does overparametrization affect performance on minority groups?cited by 2
  22. 2023Reweighted Mixup for Subpopulation Shiftcited by 2
  23. 2024Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noisecited by 2
  24. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shiftscited by 2
  25. 2025Advancing Loss Functions in Recommender Systems: A Comparative Study with a R'enyi Divergence-Based Solutioncited by 2
  26. 2023An Operational Perspective to Fairness Interventions: Where and How to Intervenecited by 1
  27. 2023Responsible AI (RAI) Games and Ensemblescited by 1
  28. 2024Cross-modality debiasing: using language to mitigate sub-population shifts in imagingcited by 1
  29. 2024Taking a Moment for Distributional Robustnesscited by 1
  30. 2024Towards Harmless Rawlsian Fairness Regardless of Demographic Priorcited by 1
  31. 2025Distributionally Robust Optimization with Adversarial Data Contaminationcited by 1
  32. 2025Distributionally Robust Federated Learning with Outlier Resiliencecited by 1
  33. 2022Diversity Boosted Learning for Domain Generalization with Large Number of Domainscited by 0
  34. 2023Robust variance-regularized risk minimization with concomitant scalingcited by 0
  35. 2024An integrated perspective of robustness in regression through the lens of the bias-variance trade-offcited by 0
  36. 2024Making Robust Generalizers Less Rigid with Loss Concentrationcited by 0
  37. 2024Alpha and Prejudice: Improving α-sized Worst-case Fairness via Intrinsic Reweightingcited by 0
  38. 2025Elliptic Loss Regularizationcited by 0

50 most cited of 56 All 56 (JSON)