Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Distributionally robust offline reinforcement learning (RL) aims to find a policy that performs the best under the worst environment within an uncertainty set using an offline dataset collected from a nominal model. While recent advances in robust RL focus on Markov decision processes (MDPs), robust non-Markovian RL is limited to planning problem where the transitions in the uncertainty set are known. In this paper, we study the learning problem of robust offline non-Markovian RL. Specifically, when the nominal model admits a low-rank structure, we propose a new algorithm, featuring a novel dataset distillation and a lower confidence bound (LCB) design for robust values under different types of the uncertainty set. We also derive new dual forms for these robust values in non-Markovian RL, making our algorithm more amenable to practical implementation. By further introducing a novel type-I concentrability coefficient tailored for offline low-rank non-Markovian decision processes, we prove that our algorithm can find an ε-optimal robust policy using O(1/ε2) offline samples. Moreover, we extend our algorithm to the case when the nominal model does not have specific structure. With a new type-II concentrability coefficient, the extended algorithm also enjoys polynomial sample efficiency under all different types of the uncertainty set.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 71 references that are in the map. First posted on arXiv 12 Nov 2024.
Builds on
35 of 71 references.
Closest ancestors · 12
Same direction, newest first
- 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage16 May
- 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation23 Nov
- 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game31 May
- 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
- 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
- 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020MOPO: Model-based Offline Policy Optimization27 May
- 2020MOReL : Model-Based Offline Reinforcement Learning12 May
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
Other references · 23
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
- 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
- 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
- 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
- 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161
- 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
- 2022Robust Reinforcement Learning using Offline Datacited by 96
- 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
- 2017Reinforcement Learning under Model Mismatchcited by 52
- 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 46
- 2022PAC Reinforcement Learning for Predictive State Representationscited by 43
11 more
- 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 40
- 2022Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systemscited by 35
- 2015Distributionally Robust Counterpart in Markov Decision Processescited by 33
- 2022Partially Observable RL with B-Stability: Unified Structural Condition and Sharp Sample-Efficient Algorithmscited by 24
- 2017End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradientcited by 23
- 2023Robust Average-Reward Markov Decision Processescited by 22
- 2023Model-Free Robust Average-Reward Reinforcement Learningcited by 20
- 2022Online Policy Optimization for Robust MDPcited by 19
- 2019Distributionally Robust Partially Observable Markov Decision Process with Moment-based Ambiguitycited by 15
- 2023Provably Efficient UCB-type Algorithms For Learning Predictive State Representationscited by 6
- 2024Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Riskcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
- 2024Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learning
- 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes
- 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
- 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
- 2025Towards Optimal Offline Reinforcement Learning
- 2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation
- 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms