mlsanityData to 7 Oct 2026

Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning

arXiv 2103.10369 ↗First posted 18 Mar 2021cs.LG cs.AI stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

In real-world tasks, reinforcement learning (RL) agents frequently encounter situations that are not present during training time. To ensure reliable performance, the RL agents need to exhibit robustness against worst-case situations. The robust RL framework addresses this challenge via a worst-case optimization between an agent and an adversary. Previous robust RL algorithms are either sample inefficient, lack robustness guarantees, or do not scale to large problems. We propose the Robust Hallucinated Upper-Confidence RL (RH-UCRL) algorithm to provably solve this problem while attaining near-optimal sample complexity guarantees. RH-UCRL is a model-based reinforcement learning (MBRL) algorithm that effectively distinguishes between epistemic and aleatoric uncertainty and efficiently explores both the agent and adversary decision spaces during policy learning. We scale RH-UCRL to complex tasks via neural networks ensemble models as well as neural network policies. Experimentally, we demonstrate that RH-UCRL outperforms other robust deep RL algorithms in a variety of adversarial environments.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

26 of 53 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  8. 2016Deep Exploration via Bootstrapped DQNcited by 518
  9. 2017Robust Adversarial Reinforcement Learningcited by 439
  10. 2017On Kernelized Multi-armed Banditscited by 348
  11. 2019Challenges of Real-World Reinforcement Learningcited by 282
  12. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
14 more
  1. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
  2. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  3. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  4. 2020Provable Self-Play Algorithms for Competitive Reinforcement Learningcited by 112
  5. 2020Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexitycited by 91
  6. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  7. 2016Learning values across many orders of magnitudecited by 80
  8. 2020Robust Reinforcement Learning using Adversarial Populationscited by 64
  9. 2020Distributionally Robust Bayesian Optimizationcited by 60
  10. 2017Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Controlcited by 50
  11. 2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
  12. 2019On Online Learning in Kernelized Markov Decision Processescited by 32
  13. 2019Calibrated Model-Based Deep Reinforcement Learningcited by 31
  14. 2017Streaming kernel regression with provably adaptive mean, variance, and regularizationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 2; 2025: 2; 2026: 2.

14 papers.

Sort:

By area:

  1. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  2. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  3. 2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
  4. 2023Plan To Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learningcited by 16
  5. 2022Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computationcited by 14
  6. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  7. 2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
  8. 2022Safe Reinforcement Learning via Confidence-Based Filterscited by 5
  9. 2024Adversarially Robust Decision Transformercited by 4
  10. 2024Towards Unraveling and Improving Generalization in World Modelscited by 3
  11. 2026Golden Handcuffs make safer AI agentscited by 1
  12. 2022User-Oriented Robust Reinforcement Learningcited by 0
2 more
  1. 2023Episodic Gaussian Process-Based Learning Control with Vanishing Tracking Errorscited by 0
  2. 2026Optimistic Policy Learning under Pessimistic Adversaries with Regret and Violation Guaranteescited by 0