mlsanityData to 7 Oct 2026

Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning

arXiv 2605.22376 ↗First posted 21 May 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Cross-domain offline reinforcement learning (CDRL) aims to improve policy learning in a target domain by leveraging data collected from a source domain. Existing works typically assess the transferability of source-domain data by measuring its similarity to target-domain transitions, and implicitly perform transition-level selection. Transitions that are considered similar are assigned higher weights or rewards, while dissimilar ones are down-weighted. However, transition-level similarity does not necessarily imply consistency in long-term returns. Even visually or dynamically similar transitions may lead to significantly different outcomes in the target domain, which can mislead policy learning and degrade performance. To address this issue, we revisit the fundamental objective of policy learning. Since policy optimization ultimately relies on Bellman targets to evaluate the quality of decisions, we propose to assess the transferability of source-domain transitions based on their alignment with target-domain Bellman targets, rather than superficial transition similarity. Based on this insight, we propose a method termed Target-Aligned Bellman Backup (TABB), which selectively leverages source-domain data by measuring their contribution to accurate Bellman target estimation in the target domain. We evaluate TABB across a broad range of cross-domain offline RL settings with highly limited target-domain data. Experimental results show that TABB consistently achieves strong performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 34 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

26 of 34 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts2 Dec
  2. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learning28 Jul
  3. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning10 Jun
  4. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone9 Dec
  5. 2024Offline Reinforcement Learning with Domain-Unlabeled Data11 Apr
  6. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning22 Jun
  7. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning13 Mar
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  7. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  8. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  9. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  10. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 30
  11. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  12. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
2 more
  1. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendationcited by 18
  2. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correction
  2. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learning
  3. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering
  4. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning
  5. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning
  6. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets
  7. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
  8. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning