mlsanityData to 14 Oct 2026

Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage

arXiv 2305.09659 ↗First posted 16 May 2023cs.LG cs.AI math.OCcited by 51

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In this paper, we study distributionally robust offline reinforcement learning (robust offline RL), which seeks to find an optimal policy purely from an offline dataset that can perform well in perturbed environments. In specific, we propose a generic algorithm framework called Doubly Pessimistic Model-based Policy Optimization (P2MPO), which features a novel combination of a flexible model estimation subroutine and a doubly pessimistic policy optimization step. Notably, the double pessimism principle is crucial to overcome the distributional shifts incurred by (i) the mismatch between the behavior policy and the target policies; and (ii) the perturbation of the nominal model. Under certain accuracy conditions on the model estimation subroutine, we prove that P2MPO is sample-efficient with robust partial coverage data, which only requires the offline data to have good coverage of the distributions induced by the optimal robust policy and the perturbed models around the nominal model. By tailoring specific model estimation subroutines for concrete examples of RMDPs, including tabular RMDPs, factored RMDPs, kernel and neural RMDPs, we prove that P2MPO enjoys a tildeO(n-1/2) convergence rate, where n is the dataset size. We highlight that all these examples, except tabular RMDPs, are first identified and proven tractable by this work. Furthermore, we continue our study of robust offline RL in the robust Markov games (RMGs). By extending the double pessimism principle identified for single-agent RMDPs, we propose another algorithm framework that can efficiently find the robust Nash equilibria among players using only robust unilateral (partial) coverage data. To our best knowledge, this work proposes the first general learning principle -- double pessimism -- for robust offline RL and shows that it is provably efficient with general function approximation.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 96 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

50 of 96 references.

Closest ancestors · 21

Same direction, newest first

  1. 2023Offline Learning in Markov Games with General Function Approximation6 Feb
  2. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian1 Nov
  3. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation14 Sep
  4. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  5. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonus1 Jun
  6. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game31 May
  7. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes26 May
  8. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  9. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  10. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  11. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  12. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
9 more
  1. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  2. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  4. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  6. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 29

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018Learning Dexterous In-Hand Manipulationcited by 687
  3. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  5. 2017Robust Adversarial Reinforcement Learningcited by 451
  6. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  8. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  9. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  10. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  11. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  12. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
17 more
  1. 2022Robust Reinforcement Learning using Offline Datacited by 96
  2. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
  3. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  4. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  5. 2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 52
  6. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 44
  7. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  8. 2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 36
  9. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 28
  10. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  11. 2023Towards Minimax Optimality of Model-based Robust Reinforcement Learningcited by 22
  12. 2020Distributionally Robust Batch Contextual Banditscited by 20
  13. 2022Online Policy Optimization for Robust MDPcited by 19
  14. 2021Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimizationcited by 18
  15. 2016Supervision via Competition: Robot Adversaries for Learning Taskscited by 16
  16. 2022Provable Sim-to-real Transfer in Continuous Domain with Partial Observationscited by 9
  17. 2022Learning Two-Player Mixture Markov Games: Kernel Function Approximation and Correlated Equilibriumcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 17; 2025: 16; 2026: 8.

51 papers.

Sort:

By area:

  1. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  2. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 52
  3. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 36
  4. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 26
  5. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  6. 2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximationcited by 25
  7. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 25
  8. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  9. 2023Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Explorationcited by 20
  10. 2023On the Foundation of Distributionally Robust Reinforcement Learningcited by 18
  11. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 18
  12. 2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
38 more
  1. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 16
  2. 2024Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learningcited by 16
  3. 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processescited by 14
  4. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
  5. 2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularizationcited by 8
  6. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 6
  7. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  8. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  9. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Datacited by 4
  10. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 4
  11. 2025Learning an Optimal Assortment Policy under Observational Datacited by 4
  12. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  13. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approachcited by 3
  14. 2023Posterior Sampling for Competitive RL: Function Approximation and Partial Observationcited by 3
  15. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  16. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruptioncited by 3
  17. 2024Leveraging Unlabeled Data Sharing through Kernel Function Approximation in Offline Reinforcement Learningcited by 2
  18. 2025Online Robust Reinforcement Learning with General Function Approximationcited by 2
  19. 2026Robust Adversarial Policy Optimization Under Dynamics Uncertaintycited by 2
  20. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  21. 2025Near-Optimal Sample Complexities of Divergence-based S-rectangular Distributionally Robust Reinforcement Learningcited by 1
  22. 2025Online Bayesian Risk-Averse Reinforcement Learningcited by 1
  23. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RLcited by 1
  24. 2025Distributionally Robust Self Paced Curriculum Reinforcement Learningcited by 1
  25. 2025Distributionally Robust Online Markov Game with Linear Function Approximationcited by 1
  26. 2025Bayesian Ambiguity Contraction-based Adaptive Robust Markov Decision Processes for Adversarial Surveillance Missionscited by 1
  27. 2024Efficient Duple Perturbation Robustness in Low-rank MDPscited by 0
  28. 2024Learning Optimal Distributionally Robust Stochastic Control in Continuous State Spacescited by 0
  29. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processescited by 0
  30. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0
  31. 2025Robust Exploratory Stopping under Ambiguity in Reinforcement Learningcited by 0
  32. 2025Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Setscited by 0
  33. 2026Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processescited by 0
  34. 2026Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorizationcited by 0
  35. 2026Taming the Curses of Multiagency in Robust Markov Games with Large State Space through Linear Function Approximationcited by 0
  36. 2026Evolving Robustness--Exploration Trade-off in Online Reinforcement Learning via Quantile Bayesian Risk MDPscited by 0
  37. 2026Robust Q-learning for mean-field control under Wasserstein uncertainty in common noisecited by 0
  38. 2026Stationary Robust Mean-Field Games under Model Mismatchescited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021DROMO: Distributionally Robust Offline Model-based Policy Optimization
  2. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  3. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
  4. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
  5. 2021COMBO: Conservative Offline Model-Based Policy Optimization
  6. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
  7. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  8. 2025Semi-pessimistic Reinforcement Learning