COSBO: Conservative Offline Simulation-Based Policy Optimization
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline reinforcement learning allows training reinforcement learning models on data from live deployments. However, it is limited to choosing the best combination of behaviors present in the training data. In contrast, simulation environments attempting to replicate the live environment can be used instead of the live data, yet this approach is limited by the simulation-to-reality gap, resulting in a bias. In an attempt to get the best of both worlds, we propose a method that combines an imperfect simulation environment with data from the target environment, to train an offline reinforcement learning policy. Our experiments demonstrate that the proposed method outperforms state-of-the-art approaches CQL, MOPO, and COMBO, especially in scenarios with diverse and challenging dynamics, and demonstrates robust behavior across a variety of experimental conditions. The results highlight that using simulator-generated data can effectively enhance offline policy learning despite the sim-to-real gap, when direct interaction with the real-world is not possible.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 24 references that are in the map. First posted on arXiv 22 Sep 2024.
Builds on
19 of 24 references.
Closest ancestors · 9
Same direction, newest first
- 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
- 2021PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators13 Feb
- 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020MOPO: Model-based Offline Policy Optimization27 May
- 2020MOReL : Model-Based Offline Reinforcement Learning12 May
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
- 2019Behavior Regularized Offline Reinforcement Learning26 Nov
Other references · 10
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
- 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
- 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
- 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
- 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
- 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
- 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
- 2020Learning predictive representations in autonomous driving to improve deep reinforcement learningcited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithm
- 2024Offline Trajectory Optimization for Offline Reinforcement Learning
- 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning
- 2025OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learning
- 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
- 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
- 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
- 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators