# Cross-Domain Policy Adaptation by Capturing Representation Mismatch

arXiv: https://arxiv.org/abs/2405.15369 · HTML: https://mlsanity.com/paper/2405.15369/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 74 of its 97 references that are in the map.
- First posted on arXiv 24 May 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 74)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Decoupling Representation Learning from Reinforcement Learning](https://mlsanity.com/paper/2009.08319/) (2020)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data](https://mlsanity.com/paper/1605.06432/) (2016)
- [Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping](https://mlsanity.com/paper/1709.07857/) (2017)
- [MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning](https://mlsanity.com/paper/2006.16908/) (2020)
- [Preparing for the Unknown: Learning a Universal Policy with Online System Identification](https://mlsanity.com/paper/1702.02453/) (2017)

## Built on by (most cited first, 25 of 25)

- [ODRL: A Benchmark for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.20750/) (2024)
- [Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.23450/) (2024)
- [xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editing](https://mlsanity.com/paper/2409.08687/) (2024)
- [Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data](https://mlsanity.com/paper/2505.23062/) (2025)
- [MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning](https://mlsanity.com/paper/2506.08460/) (2025)
- [Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignment](https://mlsanity.com/paper/2406.01103/) (2024)
- [DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learning](https://mlsanity.com/paper/2507.20499/) (2025)
- [Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts](https://mlsanity.com/paper/2512.02486/) (2025)
- [Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2412.15517/) (2024)
- [Hybrid Cross-domain Robust Reinforcement Learning](https://mlsanity.com/paper/2505.23003/) (2025)
- [A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation](https://mlsanity.com/paper/2503.05696/) (2025)
- [Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering](https://mlsanity.com/paper/2512.02435/) (2025)
- [Temporal Difference Learning with Constrained Initial Representations](https://mlsanity.com/paper/2602.11800/) (2026)
- [Cross-Domain Offline Policy Adaptation via Selective Transition Correction](https://mlsanity.com/paper/2602.05776/) (2026)
- [Semi-Supervised Cross-Domain Imitation Learning](https://mlsanity.com/paper/2602.10793/) (2026)
- [Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning](https://mlsanity.com/paper/2602.21072/) (2026)
- [Dual Alignment Maximin Optimization for Offline Model-based RL](https://mlsanity.com/paper/2502.00850/) (2025)
- [Bridging Dynamics Gaps via Diffusion Schr"odinger Bridge for Cross-Domain Reinforcement Learning](https://mlsanity.com/paper/2602.23737/) (2026)
- [Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics](https://mlsanity.com/paper/2603.12087/) (2026)
- [Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learning](https://mlsanity.com/paper/2605.13054/) (2026)
- [Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2605.24810/) (2026)
- [Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets](https://mlsanity.com/paper/2605.24862/) (2026)
- [Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment](https://mlsanity.com/paper/2605.27659/) (2026)
- [DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning](https://mlsanity.com/paper/2607.16090/) (2026)
- [Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning](https://mlsanity.com/paper/2610.02505/) (2026)

## Abstract (arXiv)

It is vital to learn effective policies that can be transferred to different domains with dynamics discrepancies in reinforcement learning (RL). In this paper, we consider dynamics adaptation settings where there exists dynamics mismatch between the source domain and the target domain, and one can get access to sufficient source domain data, while can only have limited interactions with the target domain. Existing methods address this problem by learning domain classifiers, performing data filtering from a value discrepancy perspective, etc. Instead, we tackle this challenge from a decoupled representation learning perspective. We perform representation learning only in the target domain and measure the representation deviations on the transitions from the source domain, which we show can be a signal of dynamics mismatch. We also show that representation deviation upper bounds performance difference of a given policy in the source domain and target domain, which motivates us to adopt representation deviation as a reward penalty. The produced representations are not involved in either policy or value function, but only serve as a reward penalizer. We conduct extensive experiments on environments with kinematic and morphology mismatch, and the results show that our method exhibits strong performance on many tasks. Our code is publicly available at https://github.com/dmksjfl/PAR.