mlsanityData to 7 Oct 2026

Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning

arXiv 2405.06192 ↗First posted 10 May 2024cs.LG cs.AIcited by 18

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Cross-domain offline reinforcement learning leverages source domain data with diverse transition dynamics to alleviate the data requirement for the target domain. However, simply merging the data of two domains leads to performance degradation due to the dynamics mismatch. Existing methods address this problem by measuring the dynamics gap via domain classifiers while relying on the assumptions of the transferability of paired domains. In this paper, we propose a novel representation-based approach to measure the domain gap, where the representation is learned through a contrastive objective by sampling transitions from different domains. We show that such an objective recovers the mutual-information gap of transition functions in two domains without suffering from the unbounded issue of the dynamics gap in handling significantly different domains. Based on the representations, we introduce a data filtering algorithm that selectively shares transitions from the source domain according to the contrastive score functions. Empirical results on various tasks demonstrate that our method achieves superior performance, using only 10% of the target data to achieve 89.2% of the performance on 100% target dataset with state-of-the-art methods.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 53 of its 61 references that are in the map. First posted on arXiv 10 May 2024.

Builds on

53 of 61 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning30 Apr
  2. 2024Offline Multi-task Transfer RL with Representational Penalization19 Feb
  3. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness29 Sep
  4. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps22 Sep
  5. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning22 Jun
  6. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learning10 Apr
  7. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  8. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  9. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  10. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  11. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  12. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
7 more
  1. 2021False Correlation Reduction for Offline Reinforcement Learning24 Oct
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
  4. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning16 Sep
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 34

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  9. 2019On Variational Bounds of Mutual Informationcited by 550
  10. 2020Reinforcement Learning with Augmented Datacited by 452
  11. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  12. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
22 more
  1. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  2. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  3. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  4. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
  5. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  6. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  7. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  8. 2023Finetuning Offline World Models in the Real Worldcited by 39
  9. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  10. 2020IDDA: a large-scale multi-domain dataset for autonomous drivingcited by 31
  11. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 30
  12. 2022Robust Imitation Learning against Variations in Environment Dynamicscited by 26
  13. 2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
  14. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  15. 2021Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimizationcited by 18
  16. 2022Semi-Markov Offline Reinforcement Learning for Healthcarecited by 18
  17. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 18
  18. 2023Robust Quadrupedal Locomotion via Risk-Averse Policy Learningcited by 16
  19. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
  20. 2022Latent Variable Representation for Reinforcement Learningcited by 14
  21. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  22. 2021Synthetic Acute Hypotension and Sepsis Datasets Based on MIMIC-III and Published as Part of the Health Gym Projectcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 8.

18 papers.

Sort:

By area:

  1. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  2. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  3. 2024Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Datacited by 8
  4. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Datacited by 8
  5. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 8
  6. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  7. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  8. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  9. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  10. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filteringcited by 2
  11. 2026DADP: Domain Adaptive Diffusion Policycited by 1
  12. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correctioncited by 1
6 more
  1. 2026Semi-Supervised Cross-Domain Imitation Learningcited by 1
  2. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learningcited by 1
  3. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  4. 2026Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learningcited by 0
  5. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0
  6. 2026DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples
  2. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts
  3. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  4. 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning
  5. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning
  6. 2024Integrating Domain Knowledge for handling Limited Data in Offline RL
  7. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationarity
  8. 2024Contrastive Diffuser: Planning Towards High Return States via Contrastive Learning