mlsanityData to 7 Oct 2026

Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts

arXiv 2512.02486 ↗First posted 2 Dec 2025cs.LGcited by 5

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Single-domain offline reinforcement learning (RL) often suffers from limited data coverage, while cross-domain offline RL handles this issue by leveraging additional data from other domains with dynamics shifts. However, existing studies primarily focus on train-time robustness (handling dynamics shifts from training data), neglecting the test-time robustness against dynamics perturbations when deployed in practical scenarios. In this paper, we investigate dual (both train-time and test-time) robustness against dynamics shifts in cross-domain offline RL. We first empirically show that the policy trained with cross-domain offline RL exhibits fragility under dynamics perturbations during evaluation, particularly when target domain data is limited. To address this, we introduce a novel robust cross-domain Bellman (RCB) operator, which enhances test-time robustness against dynamics perturbations while staying conservative to the out-of-distribution dynamics transitions, thus guaranteeing the train-time robustness. To further counteract potential value overestimation or underestimation caused by the RCB operator, we introduce two techniques, the dynamic value penalty and the Huber loss, into our framework, resulting in the practical Dual-RObust Cross-domain Offline RL (DROCO) algorithm. Extensive empirical results across various dynamics shift scenarios show that DROCO outperforms strong baselines and exhibits enhanced robustness to dynamics perturbations.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 57 of its 80 references that are in the map. First posted on arXiv 2 Dec 2025.

Builds on

57 of 80 references.

Closest ancestors · 30

Same direction, newest first

  1. 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
  2. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions1 Nov
  3. 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning23 Aug
  4. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning10 Jul
  5. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling5 Jul
  6. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning30 May
  7. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning14 Mar
  8. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator7 Dec
  9. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximation23 Oct
  10. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption19 Oct
  11. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps22 Sep
  12. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning22 Jun
18 more
  1. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning11 Jun
  2. 2023Survival Instinct in Offline Reinforcement Learning5 Jun
  3. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  4. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  5. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  6. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  7. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  8. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  9. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning13 Mar
  10. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  13. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  14. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  15. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  16. 2020MOPO: Model-based Offline Policy Optimization27 May
  17. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  18. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  8. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  9. 2020Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistencycited by 43
  10. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  11. 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
  12. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 30
15 more
  1. 2023RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learningcited by 29
  2. 2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximationcited by 25
  3. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  4. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
  5. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  6. 2021Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimizationcited by 18
  7. 2021On Empirical Risk Minimization with Dependent and Heavy-Tailed Datacited by 16
  8. 2023Robust Quadrupedal Locomotion via Risk-Averse Policy Learningcited by 16
  9. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  10. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  11. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  12. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Datacited by 8
  13. 2025Sample Complexity of Distributionally Robust Off-Dynamics Reinforcement Learning with Online Interactioncited by 7
  14. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  15. 2025TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-makingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 2
  2. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  3. 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learningcited by 0
  4. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0
  5. 2026Adaptive Safety Filtering for Frozen ACC Policies via Conformal Residual Calibrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering
  2. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  3. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples
  4. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correction
  5. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  6. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning
  7. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning
  8. 2021Instabilities of Offline RL with Pre-Trained Neural Representation