# Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning

arXiv: https://arxiv.org/abs/2405.06192 · HTML: https://mlsanity.com/paper/2405.06192/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 53 of its 61 references that are in the map.
- First posted on arXiv 10 May 2024.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 30 of 53)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees](https://mlsanity.com/paper/1807.03858/) (2018)
- [Representation Learning for Online and Offline RL in Low-rank MDPs](https://mlsanity.com/paper/2110.04652/) (2021)
- [Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning](https://mlsanity.com/paper/2202.11566/) (2022)
- [Offline Reinforcement Learning with Realizability and Single-policy Concentrability](https://mlsanity.com/paper/2202.04634/) (2022)
- [Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion](https://mlsanity.com/paper/2311.01017/) (2023)
- [Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers](https://mlsanity.com/paper/2006.13916/) (2020)
- [RORL: Robust Offline Reinforcement Learning via Conservative Smoothing](https://mlsanity.com/paper/2206.02829/) (2022)
- [Supported Policy Optimization for Offline Reinforcement Learning](https://mlsanity.com/paper/2202.06239/) (2022)
- [Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task](https://mlsanity.com/paper/1707.02267/) (2017)
- [Conservative Data Sharing for Multi-Task Offline Reinforcement Learning](https://mlsanity.com/paper/2109.08128/) (2021)
- [How to Leverage Unlabeled Data in Offline Reinforcement Learning](https://mlsanity.com/paper/2202.01741/) (2022)
- [When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning](https://mlsanity.com/paper/2206.13464/) (2022)

## Built on by (most cited first, 18 of 18)

- [ODRL: A Benchmark for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.20750/) (2024)
- [Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.23450/) (2024)
- [Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data](https://mlsanity.com/paper/2505.23062/) (2025)
- [MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning](https://mlsanity.com/paper/2506.08460/) (2025)
- [Constrained Ensemble Exploration for Unsupervised Skill Discovery](https://mlsanity.com/paper/2405.16030/) (2024)
- [Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Data](https://mlsanity.com/paper/2411.03810/) (2024)
- [DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learning](https://mlsanity.com/paper/2507.20499/) (2025)
- [Linear Mixture Distributionally Robust Markov Decision Processes](https://mlsanity.com/paper/2505.18044/) (2025)
- [Hybrid Cross-domain Robust Reinforcement Learning](https://mlsanity.com/paper/2505.23003/) (2025)
- [Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering](https://mlsanity.com/paper/2512.02435/) (2025)
- [DADP: Domain Adaptive Diffusion Policy](https://mlsanity.com/paper/2602.04037/) (2026)
- [Cross-Domain Offline Policy Adaptation via Selective Transition Correction](https://mlsanity.com/paper/2602.05776/) (2026)
- [Semi-Supervised Cross-Domain Imitation Learning](https://mlsanity.com/paper/2602.10793/) (2026)
- [Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning](https://mlsanity.com/paper/2602.21072/) (2026)
- [Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learning](https://mlsanity.com/paper/2605.13054/) (2026)
- [Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2605.24810/) (2026)
- [Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets](https://mlsanity.com/paper/2605.24862/) (2026)
- [DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning](https://mlsanity.com/paper/2607.16090/) (2026)

## Abstract (arXiv)

Cross-domain offline reinforcement learning leverages source domain data with diverse transition dynamics to alleviate the data requirement for the target domain. However, simply merging the data of two domains leads to performance degradation due to the dynamics mismatch. Existing methods address this problem by measuring the dynamics gap via domain classifiers while relying on the assumptions of the transferability of paired domains. In this paper, we propose a novel representation-based approach to measure the domain gap, where the representation is learned through a contrastive objective by sampling transitions from different domains. We show that such an objective recovers the mutual-information gap of transition functions in two domains without suffering from the unbounded issue of the dynamics gap in handling significantly different domains. Based on the representations, we introduce a data filtering algorithm that selectively shares transitions from the source domain according to the contrastive score functions. Empirical results on various tasks demonstrate that our method achieves superior performance, using only 10% of the target data to achieve 89.2% of the performance on 100% target dataset with state-of-the-art methods.