mlsanityData to 7 Oct 2026

Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning

arXiv 2306.12755 ↗First posted 22 Jun 2023cs.LGcited by 27

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) aims to learn a policy using only pre-collected and fixed data. Although avoiding the time-consuming online interactions in RL, it poses challenges for out-of-distribution (OOD) state actions and often suffers from data inefficiency for training. Despite many efforts being devoted to addressing OOD state actions, the latter (data inefficiency) receives little attention in offline RL. To address this, this paper proposes the cross-domain offline RL, which assumes offline data incorporate additional source-domain data from varying transition dynamics (environments), and expects it to contribute to the offline data efficiency. To do so, we identify a new challenge of OOD transition dynamics, beyond the common OOD state actions issue, when utilizing cross-domain offline data. Then, we propose our method BOSA, which employs two support-constrained objectives to address the above OOD issues. Through extensive experiments in the cross-domain offline RL setting, we demonstrate BOSA can greatly improve offline data efficiency: using only 10% of the target data, BOSA could achieve 74.4% of the SOTA offline RL performance that uses 100% of the target data. Additionally, we also show BOSA can be effortlessly plugged into model-based offline RL and noising data augmentation techniques (used for generating source-domain data), which naturally avoids the potential dynamics mismatch between target-domain data and newly generated source-domain data.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 50 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

42 of 50 references.

Closest ancestors · 22

Same direction, newest first

  1. 2023Beyond Reward: Offline Preference-guided Policy Optimization25 May
  2. 2023Behavior Proximal Policy Optimization22 Feb
  3. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning14 Jun
  4. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  5. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning13 Mar
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  7. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  11. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
  12. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
10 more
  1. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  2. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020MOPO: Model-based Offline Policy Optimization27 May
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 20

  1. 2015Trust Region Policy Optimizationcited by 2,503
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2019A Theory of Regularized Markov Decision Processescited by 279
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  8. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  9. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  10. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  11. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  12. 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49
8 more
  1. 2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
  2. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  3. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  4. 2017Mutual Alignment Transfer Learningcited by 16
  5. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 15
  6. 2021Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learningcited by 15
  7. 2022Learn what matters: cross-domain imitation learning with task-relevant embeddingscited by 14
  8. 2021Off-Dynamics Inverse Reinforcement Learning from Hetero-Domaincited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 6; 2025: 8; 2026: 8.

27 papers.

Sort:

By area:

  1. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  2. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  3. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  4. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  5. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  6. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systemscited by 10
  7. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  8. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 8
  9. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  10. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  11. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  12. 2024Offline Reinforcement Learning with Domain-Unlabeled Datacited by 3
15 more
  1. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  2. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filteringcited by 2
  3. 2024Random Policy Enables In-Context Reinforcement Learning within Trust Horizonscited by 1
  4. 2025Boundary-to-Region Supervision for Offline Safe Reinforcement Learningcited by 1
  5. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correctioncited by 1
  6. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learningcited by 1
  7. 2023Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Explorationcited by 0
  8. 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RLcited by 0
  9. 2025Target-Aligned Fusion for Decision-Sequence Learning under Dynamics Shiftcited by 0
  10. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  11. 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learningcited by 0
  12. 2026Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learningcited by 0
  13. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0
  14. 2026Cross-Domain Off-Policy Evaluation and Learning for Contextual Banditscited by 0
  15. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Augmenting Offline RL with Unlabeled Data
  2. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
  3. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  4. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  5. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  6. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  7. 2022Leveraging Offline Data in Online Reinforcement Learning
  8. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows