Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline reinforcement learning (RL) methods strike a balance between exploration and exploitation by conservative value estimation -- penalizing values of unseen states and actions. Model-free methods penalize values at all unseen actions, while model-based methods are able to further exploit unseen states via model rollouts. However, such methods are handicapped in their ability to find unseen states far away from the available offline data due to two factors -- (a) very short rollout horizons in models due to cascading model errors, and (b) model rollouts originating solely from states observed in offline data. We relax the second assumption and present a novel unseen state augmentation strategy to allow exploitation of unseen states where the learned model and value estimates generalize. Our strategy finds unseen states by value-informed perturbations of seen states followed by filtering out states with epistemic uncertainty estimates too high (high error) or too low (too similar to seen data). We observe improved performance in several offline RL tasks and find that our augmentation strategy consistently leads to overall lower average dataset Q-value estimates i.e. more conservative Q-value estimates than a baseline.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 16 of its 19 references that are in the map. First posted on arXiv 7 Aug 2023.
Builds on
16 of 19 references.
Closest ancestors · 9
Same direction, newest first
- 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
- 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
- 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
- 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020MOPO: Model-based Offline Policy Optimization27 May
- 2020MOReL : Model-Based Offline Reinforcement Learning12 May
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
Other references · 7
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
- 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
- 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
- 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
- 2021Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentationcited by 15
- 2022Revisiting Model-based Value Expansioncited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2021Offline Reinforcement Learning with Implicit Q-Learning
- 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation
- 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
- 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning
- 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
- 2023Equivariant Data Augmentation for Generalization in Offline Reinforcement Learning
- 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
- 2024Augmenting Offline Reinforcement Learning with State-only Interactions