Robust Reinforcement Learning with Wasserstein Constraint
Not on the map. Why?
Abstract (arXiv)
Robust Reinforcement Learning aims to find the optimal policy with some extent of robustness to environmental dynamics. Existing learning algorithms usually enable the robustness through disturbing the current state or simulating environmental parameters in a heuristic way, which lack quantified robustness to the system dynamics (i.e. transition probability). To overcome this issue, we leverage Wasserstein distance to measure the disturbance to the reference transition kernel. With Wasserstein distance, we are able to connect transition kernel disturbance to the state disturbance, i.e. reduce an infinite-dimensional optimization problem to a finite-dimensional risk-aware problem. Through the derived risk-aware optimal Bellman equation, we show the existence of optimal robust policies, provide a sensitivity analysis for the perturbations, and then design a novel robust learning algorithm--Wasserstein Robust Advantage Actor-Critic algorithm (WRAAC). The effectiveness of the proposed algorithm is verified in the Cart-Pole environment.
Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jun 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 3; 2022: 4; 2023: 4; 2024: 1; 2025: 5; 2026: 1.
18 papers.
Sort:
By area:
- 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
- 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
- 2023Robust Average-Reward Markov Decision Processescited by 22
- 2021Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimizationcited by 18
- 2022Distributionally Robust Bayesian Optimization with varphi-divergencescited by 16
- 2022Robust Constrained Reinforcement Learningcited by 14
- 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9
- 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
- 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 5
- 2025Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learningcited by 4
- 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
- 2022Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learningcited by 2
6 more
- 2025Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mappingcited by 2
- 2021Learning to Play General-Sum Games Against Multiple Boundedly Rational Agentscited by 1
- 2024How does Your RL Agent Explore? An Optimal Transport Analysis of Occupancy Measure Trajectoriescited by 1
- 2025Online Robust Reinforcement Learning with General Function Approximationcited by 1
- 2023On Practical Robust Reinforcement Learning: Practical Uncertainty Set and Double-Agent Algorithmcited by 0
- 2026Counterfactually Safe Reinforcement Learningcited by 0
- 2026Counterfactually Safe Reinforcement Learning24 May 2026
- 2025Online Robust Reinforcement Learning with General Function Approximation22 Dec 2025
- 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality24 Aug 2025
- 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning5 Aug 2025
- 2025Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping12 Mar 2025
- 2025Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learning17 Jan 2025
- 2024How does Your RL Agent Explore? An Optimal Transport Analysis of Occupancy Measure Trajectories14 Feb 2024
- 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator7 Dec 2023
- 2023On Practical Robust Reinforcement Learning: Practical Uncertainty Set and Double-Agent Algorithm11 May 2023
- 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guarantees31 Jan 2023
- 2023Robust Average-Reward Markov Decision Processes2 Jan 2023
- 2022Robust Constrained Reinforcement Learning14 Sep 2022
6 more
- 2022Policy Gradient Method For Robust Reinforcement Learning15 May 2022
- 2022Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning22 Mar 2022
- 2022Distributionally Robust Bayesian Optimization with varphi-divergences4 Mar 2022
- 2021Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization20 Dec 2021
- 2021Online Robust Reinforcement Learning with Model Uncertainty29 Sep 2021
- 2021Learning to Play General-Sum Games Against Multiple Boundedly Rational Agents10 Jun 2021
- 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
- 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
- 2023Robust Average-Reward Markov Decision Processescited by 22
- 2021Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimizationcited by 18
- 2022Distributionally Robust Bayesian Optimization with varphi-divergencescited by 16
- 2022Robust Constrained Reinforcement Learningcited by 14
- 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9
- 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 5
- 2025Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learningcited by 4
- 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
- 2022Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learningcited by 2
- 2025Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mappingcited by 2
12 most cited of 17 All 17 (JSON)
- 2026Counterfactually Safe Reinforcement Learning24 May 2026
- 2025Online Robust Reinforcement Learning with General Function Approximation22 Dec 2025
- 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality24 Aug 2025
- 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning5 Aug 2025
- 2025Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping12 Mar 2025
- 2025Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learning17 Jan 2025
- 2024How does Your RL Agent Explore? An Optimal Transport Analysis of Occupancy Measure Trajectories14 Feb 2024
- 2023On Practical Robust Reinforcement Learning: Practical Uncertainty Set and Double-Agent Algorithm11 May 2023
- 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guarantees31 Jan 2023
- 2023Robust Average-Reward Markov Decision Processes2 Jan 2023
- 2022Robust Constrained Reinforcement Learning14 Sep 2022
- 2022Policy Gradient Method For Robust Reinforcement Learning15 May 2022
12 newest of 17 All 17 (JSON)