mlsanityData to 14 Oct 2026

First-Order Methods for Wasserstein Distributionally Robust MDP

arXiv 2009.06790 ↗First posted 14 Sep 2020math.OC cs.GTcited by 17

Not on the map. Why?

Abstract (arXiv)

Markov decision processes (MDPs) are known to be sensitive to parameter specification. Distributionally robust MDPs alleviate this issue by allowing for ambiguity sets which give a set of possible distributions over parameter sets. The goal is to find an optimal policy with respect to the worst-case parameter distribution. We propose a framework for solving Distributionally robust MDPs via first-order methods, and instantiate it for several types of Wasserstein ambiguity sets. By developing efficient proximal updates, our algorithms achieve a convergence rate of Oleft(NA2.5S3.5log(S)log(ε-1)ε-1.5 right) for the number of kernels N in the support of the nominal distribution, states S, and actions A; this rate varies slightly based on the Wasserstein setup. Our dependence on N,A and S is significantly better than existing methods, which have a complexity of Oleft(N3.5A3.5S4.5log2(ε-1) right). Numerical experiments show that our algorithm is significantly more scalable than state-of-the-art approaches across several domains.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 3; 2024: 2; 2025: 2; 2026: 2.

17 papers.

Sort:

By area:

  1. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 38
  2. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 10
  3. 2022Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variablescited by 7
  4. 2025Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learningcited by 7
  5. 2021Conic Blackwell Algorithm: Parameter-Free Convex-Concave Saddle-Point Solvingcited by 5
  6. 2025Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gaincited by 5
  7. 2022Solving optimization problems with Blackwell approachabilitycited by 4
  8. 2022RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Riskcited by 4
  9. 2024Relative Translation Invariant Wasserstein Distancecited by 3
  10. 2022Robust Phi-Divergence MDPscited by 1
  11. 2022User-Oriented Robust Reinforcement Learningcited by 0
  12. 2023Risk-Averse MDPs under Reward Ambiguitycited by 0
5 more
  1. 2023On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processescited by 0
  2. 2023Solving Robust MDPs through No-Regret Dynamicscited by 0
  3. 2024Percentile Criterion Optimization in Offline Reinforcement Learningcited by 0
  4. 2026A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatchcited by 0
  5. 2026Policy Gradient Learning for Distributionally Robust Markov Decision Processes under Wasserstein Ambiguitycited by 0