mlsanityData to 14 Oct 2026

Solving Multi-Model MDPs by Coordinate Ascent and Dynamic Programming

arXiv 2407.06329 ↗First posted 8 Jul 2024cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Multi-model Markov decision process (MMDP) is a promising framework for computing policies that are robust to parameter uncertainty in MDPs. MMDPs aim to find a policy that maximizes the expected return over a distribution of MDP models. Because MMDPs are NP-hard to solve, most methods resort to approximations. In this paper, we derive the policy gradient of MMDPs and propose CADP, which combines a coordinate ascent method and a dynamic programming algorithm for solving MMDPs. The main innovation of CADP compared with earlier algorithms is to take the coordinate ascent perspective to adjust model weights iteratively to guarantee monotone policy improvements to a local maximum. A theoretical analysis of CADP proves that it never performs worse than previous dynamic programming algorithms like WSU. Our numerical results indicate that CADP substantially outperforms existing methods on several benchmark problems.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 5 of its 41 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

5 of 41 references.

References · 5

  1. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 52
  2. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 50
  3. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 40
  4. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  5. 2021Thompson Sampling with a Mixture Priorcited by 16

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Risk-averse Total-reward MDPs with ERM and EVaRcited by 4
  2. 2025Risk-Averse Total-Reward Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet
  2. 2024Sequential Decision-Making under Uncertainty: A Robust MDPs review
  3. 2025Efficient Algorithms for Mitigating Uncertainty and Risk in Reinforcement Learning
  4. 2024A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processes
  5. 2025Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions
  6. 2024Policy Gradient for Robust Markov Decision Processes
  7. 2024Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm
  8. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form