mlsanityData to 14 Oct 2026

Learning Distributionally Robust Models at Scale via Composite Optimization

arXiv 2203.09607 ↗First posted 17 Mar 2022cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

To train machine learning models that are robust to distribution shifts in the data, distributionally robust optimization (DRO) has been proven very effective. However, the existing approaches to learning a distributionally robust model either require solving complex optimization problems such as semidefinite programming or a first-order method whose convergence scales linearly with the number of data samples -- which hinders their scalability to large datasets. In this paper, we show how different variants of DRO are simply instances of a finite-sum composite optimization for which we provide scalable methods. We also provide empirical results that demonstrate the effectiveness of our proposed algorithm with respect to the prior art in order to learn robust models from very large datasets.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 43 of its 71 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

43 of 71 references.

References · 43

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  3. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  4. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  5. 2019Agnostic Federated Learningcited by 456
  6. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
  7. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 364
  8. 2020Adaptive Personalized Federated Learningcited by 358
  9. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  10. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  11. 2015Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulationscited by 291
  12. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 287
31 more
  1. 2018Empirical Risk Minimization under Fairness Constraintscited by 226
  2. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 220
  3. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
  4. 2019Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learningcited by 170
  5. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  6. 2016Variance-based regularization with convex objectivescited by 155
  7. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  8. 2015Distributionally Robust Logistic Regressioncited by 126
  9. 2013Introducing LETOR 4.0 Datasetscited by 121
  10. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 107
  11. 2021Distributionally Robust Federated Averagingcited by 104
  12. 2014Stochastic Compositional Gradient Descent: Algorithms for Minimizing Compositions of Expected-Value Functionscited by 95
  13. 2013Data-Driven Robust Optimizationcited by 85
  14. 2020Robust Optimization for Fairness with Noisy Protected Groupscited by 78
  15. 2020Coping with Label Shift via Distributionally Robust Optimisationcited by 70
  16. 2019Pairwise Fairness for Ranking and Regressioncited by 58
  17. 2018Two-Player Games for Efficient Non-Convex Constrained Optimizationcited by 55
  18. 2016Accelerating Stochastic Composition Optimizationcited by 46
  19. 2017Robust Optimization for Non-Convex Objectivescited by 35
  20. 2019One Sample Stochastic Frank-Wolfecited by 32
  21. 2019A Stochastic Composite Gradient Method with Incremental Variance Reductioncited by 26
  22. 2021Fast Distributionally Robust Learning with Variance Reduced Min-Max Optimizationcited by 19
  23. 2016Finite-sum Composition Optimization via Variance Reduced Gradient Descentcited by 15
  24. 2019Stochastic Primal-Dual Algorithms with Faster Convergence than O(1/√T) for Problems without Bilinear Structurecited by 15
  25. 2017Accelerated Method for Stochastic Composition Optimization with Nonsmooth Regularizationcited by 13
  26. 2019A Simple and Effective Framework for Pairwise Deep Metric Learningcited by 13
  27. 2020Optimal Epoch Stochastic Gradient Descent Ascent Methods for Min-Max Optimizationcited by 3
  28. 2021Robust Experimentation in the Continuous Time Bandit Problemcited by 3
  29. 2015A Light Touch for Heavily Constrained SGDcited by 2
  30. 2019A General Analysis Framework of Lower Complexity Bounds for Finite-Sum Optimizationcited by 2
  31. 2019Zeroth-order Stochastic Compositional Algorithms for Risk-Aware Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2.

5 papers.

  1. 2022Distributed Distributionally Robust Optimization with Non-Convex Objectivescited by 14
  2. 2023Efficient Stochastic Approximation of Minimax Excess Risk Optimizationcited by 4
  3. 2022Wasserstein Distributionally Robust Optimization with Wasserstein Barycenterscited by 3
  4. 2022Faster Adaptive Momentum-Based Federated Methods for Distributed Composition Optimizationcited by 2
  5. 2023FedDRO: Federated Compositional Optimization for Distributionally Robust Learningcited by 1