mlsanityData to 14 Oct 2026

Hybrid Cross-domain Robust Reinforcement Learning

arXiv 2505.23003 ↗First posted 29 May 2025cs.LG cs.AIcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Robust reinforcement learning (RL) aims to learn policies that remain effective despite uncertainties in its environment, which frequently arise in real-world applications due to variations in environment dynamics. The robust RL methods learn a robust policy by maximizing value under the worst-case models within a predefined uncertainty set. Offline robust RL algorithms are particularly promising in scenarios where only a fixed dataset is available and new data cannot be collected. However, these approaches often require extensive offline data, and gathering such datasets for specific tasks in specific environments can be both costly and time-consuming. Using an imperfect simulator offers a faster, cheaper, and safer way to collect data for training, but it can suffer from dynamics mismatch. In this paper, we introduce HYDRO, the first Hybrid Cross-Domain Robust RL framework designed to address these challenges. HYDRO utilizes an online simulator to complement the limited amount of offline datasets in the non-trivial context of robust RL. By measuring and minimizing performance gaps between the simulator and the worst-case models in the uncertainty set, HYDRO employs novel uncertainty filtering and prioritized sampling to select the most relevant and reliable simulator samples. Our extensive experiments demonstrate HYDRO's superior performance over existing methods across various tasks, underscoring its potential to improve sample efficiency in offline robust RL.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 48 of its 65 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

48 of 65 references.

References · 48

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  7. 2015Prioritized Experience Replaycited by 1,099
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  9. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  11. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  12. 2017Robust Adversarial Reinforcement Learningcited by 451
36 more
  1. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 335
  2. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  3. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 233
  4. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
  5. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 149
  6. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 144
  7. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  8. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  9. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  10. 2018The Limits and Potentials of Deep Learning for Roboticscited by 104
  11. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 97
  12. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  13. 2022Robust Reinforcement Learning using Offline Datacited by 96
  14. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 87
  15. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  16. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  17. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  18. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  19. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  20. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  21. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
  22. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 32
  23. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 31
  24. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 31
  25. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  26. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 25
  27. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 23
  28. 2022Online Policy Optimization for Robust MDPcited by 19
  29. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  30. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 17
  31. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  32. 2022REvolveR: Continuous Evolutionary Models for Robot-to-robot Policy Transfercited by 13
  33. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
  34. 2024Policy Learning for Off-Dynamics RL with Deficient Supportcited by 9
  35. 2022Zero-Shot Policy Transfer with Disentangled Task Representation of Meta-Reinforcement Learningcited by 5
  36. 2023A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 18
  2. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  3. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2024RRLS : Robust Reinforcement Learning Suite
  2. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
  3. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
  4. 2024Safe and Robust Reinforcement Learning: Principles and Practice
  5. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning
  6. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms
  7. 2025Adversarial Diffusion for Robust Reinforcement Learning
  8. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning