First-order Policy Optimization for Robust Policy Evaluation
Not on the map. Why?
Abstract (arXiv)
We adopt a policy optimization viewpoint towards policy evaluation for robust Markov decision process with s-rectangular ambiguity sets. The developed method, named first-order policy evaluation (FRPE), provides the first unified framework for robust policy evaluation in both deterministic (offline) and stochastic (online) settings, with either tabular representation or generic function approximation. In particular, we establish linear convergence in the deterministic setting, and tildeO(1/ε2) sample complexity in the stochastic setting. FRPE also extends naturally to evaluating the robust state-action value function with (s, a)-rectangular ambiguity sets. We discuss the application of the developed results for stochastic policy optimization of large-scale robust MDPs.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 7 of its 25 references that are in the map. First posted on arXiv 29 Jul 2023.
Builds on
7 of 25 references.
References · 7
- 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
- 2017Reinforcement Learning under Model Mismatchcited by 52
- 2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
- 2020Simple and optimal methods for stochastic variational inequalities, II: Markovian noise and policy evaluation in reinforcement learningcited by 22
- 2022Policy Optimization over General State and Action Spacescited by 21
- 2021Accelerated and instance-optimal policy evaluation with linear function approximationcited by 18
- 2023Policy Mirror Descent Inherently Explores Action Spacecited by 11
Built on by
Citing papers per year: 2024: 4; 2025: 1; 2026: 1.
6 papers.
- 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
- 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 24
- 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
- 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processescited by 12
- 2025Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learningcited by 2
- 2026Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterizationcited by 1
- 2026Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterization11 Feb 2026
- 2025Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning8 Jun 2025
- 2024Distributionally Robust Constrained Reinforcement Learning under Strong Duality22 Jun 2024
- 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty29 Apr 2024
- 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms4 Apr 2024
- 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes19 Mar 2024