mlsanityData to 14 Oct 2026

Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics

arXiv 2603.12087 ↗First posted 12 Mar 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Cross-domain reinforcement learning (CDRL) is meant to improve the data efficiency of RL by leveraging the data samples collected from a source domain to facilitate the learning in a similar target domain. Despite its potential, cross-domain transfer in RL is known to have two fundamental and intertwined challenges: (i) The source and target domains can have distinct state space or action space, and this makes direct transfer infeasible and thereby requires more sophisticated inter-domain mappings; (ii) The transferability of a source-domain model in RL is not easily identifiable a priori, and hence CDRL can be prone to negative effect during transfer. In this paper, we propose to jointly tackle these two challenges through the lens of textitcross-domain Bellman consistency and textithybrid critic. Specifically, we first introduce the notion of cross-domain Bellman consistency as a way to measure transferability of a source-domain model. Then, we propose QAvatar, which combines the Q functions from both the source and target domains with an adaptive hyperparameter-free weight function. Through this design, we characterize the convergence behavior of QAvatar and show that QAvatar achieves reliable transfer in the sense that it effectively leverages a source-domain Q function for knowledge transfer to the target domain. Through experiments, we demonstrate that QAvatar achieves favorable transferability across various RL benchmark tasks, including locomotion and robot arm manipulation. Our code is available at https://rl-bandits-lab.github.io/Cross-Domain-RL/.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 54 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

32 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatch24 May
  2. 2024Cross Domain Policy Transfer with Effect Cycle-Consistency4 Mar
  3. 2024Recurrent Reinforcement Learning with Memoroids15 Feb

Other references · 29

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018DeepMind Control Suitecited by 956
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  6. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  7. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
  8. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  9. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  10. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  11. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
  12. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 97
17 more
  1. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 92
  2. 2017Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learningcited by 85
  3. 2018Learn What Not to Learn: Action Elimination with Deep Reinforcement Learningcited by 77
  4. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 64
  5. 2021Auto-Tuned Sim-to-Real Transfercited by 49
  6. 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49
  7. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  8. 2020Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistencycited by 43
  9. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
  10. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 31
  11. 2019Effects of padding on LSTMs and CNNscited by 22
  12. 2024FlowPG: Action-constrained Policy Gradient with Normalizing Flowscited by 16
  13. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guaranteescited by 11
  14. 2025Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPscited by 9
  15. 2018Target Transfer Q-Learning and Its Convergence Analysiscited by 6
  16. 2023Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptabilitycited by 3
  17. 2026Semi-Supervised Cross-Domain Imitation Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning
  2. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correction
  3. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering
  4. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts
  5. 2026Bridging Dynamics Gaps via Diffusion Schr"odinger Bridge for Cross-Domain Reinforcement Learning
  6. 2026DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning
  7. 2026One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control
  8. 2026Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agents