mlsanityData to 14 Oct 2026

On Practical Robust Reinforcement Learning: Practical Uncertainty Set and Double-Agent Algorithm

arXiv 2305.06657 ↗First posted 11 May 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Robust reinforcement learning (RRL) aims at seeking a robust policy to optimize the worst case performance over an uncertainty set of Markov decision processes (MDPs). This set contains some perturbed MDPs from a nominal MDP (N-MDP) that generate samples for training, which reflects some potential mismatches between training (i.e., N-MDP) and true environments. In this paper we present an elaborated uncertainty set by excluding some implausible MDPs from the existing sets. Under this uncertainty set, we develop a sample-based RRL algorithm (named ARQ-Learning) for tabular setting and characterize its finite-time error bound. Also, it is proved that ARQ-Learning converges as fast as the standard Q-Learning and robust Q-Learning while ensuring better robustness. We introduce an additional pessimistic agent which can tackle the major bottleneck for the extension of ARQ-Learning into the cases with larger or continuous state spaces. Incorporating this idea into RL algorithms, we propose double-agent algorithms for model-free RRL. Via experiments, we demonstrate the effectiveness of the proposed algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 50 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

28 of 50 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2016OpenAI Gymcited by 1,736
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  7. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  8. 2017Robust Adversarial Reinforcement Learningcited by 451
  9. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  10. 2017Adversarial Attacks on Neural Network Policiescited by 320
  11. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  12. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
16 more
  1. 2017Delving into adversarial attacks on deep policiescited by 104
  2. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  3. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  4. 2020Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reductioncited by 95
  5. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 80
  6. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 78
  7. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  8. 2020Robust Reinforcement Learning using Adversarial Populationscited by 66
  9. 2019Wasserstein Robust Reinforcement Learningcited by 54
  10. 2017Reinforcement Learning under Model Mismatchcited by 52
  11. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 40
  12. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  13. 2019A Bayesian Approach to Robust Reinforcement Learningcited by 34
  14. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 33
  15. 2020Robust Reinforcement Learning with Wasserstein Constraintcited by 21
  16. 2018Learning Robust Optionscited by 17

Built on by

Not cited yet.