mlsanityData to 7 Oct 2026

H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps

arXiv 2309.12716 ↗First posted 22 Sep 2023cs.LG cs.AI cs.ROcited by 15

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Solving real-world complex tasks using reinforcement learning (RL) without high-fidelity simulation environments or large amounts of offline data can be quite challenging. Online RL agents trained in imperfect simulation environments can suffer from severe sim-to-real issues. Offline RL approaches although bypass the need for simulators, often pose demanding requirements on the size and quality of the offline datasets. The recently emerged hybrid offline-and-online RL provides an attractive framework that enables joint use of limited offline data and imperfect simulator for transferable policy learning. In this paper, we develop a new algorithm, called H2O+, which offers great flexibility to bridge various choices of offline and online learning methods, while also accounting for dynamics gaps between the real and simulation environment. Through extensive simulation and real-world robotics experiments, we demonstrate superior performance and flexibility over advanced cross-domain online and offline RL algorithms.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 47 references that are in the map. First posted on arXiv 22 Sep 2023.

Builds on

39 of 47 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  3. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  4. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  5. 2022Leveraging Offline Data in Online Reinforcement Learning9 Nov
  6. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  7. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  8. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning13 Mar
  9. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  10. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
2 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 25

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019Solving Rubik's Cube with a Robot Handcited by 699
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  7. 2018Learning Dexterous In-Hand Manipulationcited by 643
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  9. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  12. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
13 more
  1. 2019Active Domain Randomizationcited by 161
  2. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  3. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  4. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 129
  5. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  6. 2022Neural-Fly Enables Rapid Learning for Agile Flight in Strong Windscited by 80
  7. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 78
  8. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  9. 2021Auto-Tuned Sim-to-Real Transfercited by 45
  10. 2020Data-efficient Domain Randomization with Bayesian Optimizationcited by 43
  11. 2019How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?cited by 25
  12. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  13. 2022Real-time Model Predictive Control and System Identification Using Differentiable Physics Simulationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 2.

15 papers.

Sort:

By area:

  1. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  2. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  3. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  4. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  5. 2024Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Datacited by 8
  6. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  7. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  8. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  9. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filteringcited by 2
  10. 2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimationcited by 1
  11. 2025Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modelingcited by 1
  12. 2025Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learningcited by 0
3 more
  1. 2025Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximizationcited by 0
  2. 2026Completion-Aware Cross-Fidelity Offline-to-Online Reinforcement Learning for Multi-Line Bus Holdingcited by 0
  3. 2026Calibration-risk routing for controlled world-model adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  2. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators
  3. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  5. 2026Sim2O: Efficient Offline-to-Online MARL via Joint Action Composition
  6. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots
  7. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learning
  8. 2020MOReL : Model-Based Offline Reinforcement Learning