mlsanityData to 14 Oct 2026

Cross-Domain Policy Adaptation by Capturing Representation Mismatch

arXiv 2405.15369 ↗First posted 24 May 2024cs.LG cs.AIcited by 25

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

It is vital to learn effective policies that can be transferred to different domains with dynamics discrepancies in reinforcement learning (RL). In this paper, we consider dynamics adaptation settings where there exists dynamics mismatch between the source domain and the target domain, and one can get access to sufficient source domain data, while can only have limited interactions with the target domain. Existing methods address this problem by learning domain classifiers, performing data filtering from a value discrepancy perspective, etc. Instead, we tackle this challenge from a decoupled representation learning perspective. We perform representation learning only in the target domain and measure the representation deviations on the transitions from the source domain, which we show can be a signal of dynamics mismatch. We also show that representation deviation upper bounds performance difference of a given policy in the source domain and target domain, which motivates us to adopt representation deviation as a reward penalty. The produced representations are not involved in either policy or value function, but only serve as a reward penalizer. We conduct extensive experiments on environments with kinematic and morphology mismatch, and the results show that our method exhibits strong performance on many tasks. Our code is publicly available at https://github.com/dmksjfl/PAR.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 74 of its 97 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

74 of 97 references.

References · 74

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  10. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  12. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
38 more
  1. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  2. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  3. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  4. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  5. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  6. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  7. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  8. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  9. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  10. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  11. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  12. 2021Mastering Atari Games with Limited Datacited by 237
  13. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
  14. 2019Active Domain Randomizationcited by 170
  15. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 156
  16. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 144
  17. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 142
  18. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 141
  19. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  20. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  21. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  22. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 97
  23. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 95
  24. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 85
  25. 2019Learning Action Representations for Reinforcement Learningcited by 75
  26. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  27. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  28. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 54
  29. 2019Simulation-based reinforcement learning for real-world autonomous drivingcited by 52
  30. 2018Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translationcited by 50
  31. 2019Meta Reinforcement Learning for Sim-to-real Domain Adaptationcited by 49
  32. 2021Auto-Tuned Sim-to-Real Transfercited by 49
  33. 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49
  34. 2022Understanding Self-Predictive Learning for Reinforcement Learningcited by 44
  35. 2020Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistencycited by 43
  36. 2021Cross-domain Imitation from Observationscited by 42
  37. 2019Environment Probing Interaction Policiescited by 39
  38. 2020Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?cited by 37

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 12.

25 papers.

Sort:

By area:

  1. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  2. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  3. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  4. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Datacited by 9
  5. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 9
  6. 2024Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignmentcited by 8
  7. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  8. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 6
  9. 2024Novelty-Guided Data Reuse for Efficient and Diversified Multi-Agent Reinforcement Learningcited by 4
  10. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  11. 2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimationcited by 2
  12. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filteringcited by 2
13 more
  1. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  2. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correctioncited by 1
  3. 2026Semi-Supervised Cross-Domain Imitation Learningcited by 1
  4. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learningcited by 1
  5. 2025Dual Alignment Maximin Optimization for Offline Model-based RLcited by 0
  6. 2026Bridging Dynamics Gaps via Diffusion Schr"odinger Bridge for Cross-Domain Reinforcement Learningcited by 0
  7. 2026Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Criticscited by 0
  8. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  9. 2026Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learningcited by 0
  10. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0
  11. 2026Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deploymentcited by 0
  12. 2026DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learningcited by 0
  13. 2026Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Policy Learning for Off-Dynamics RL with Deficient Support
  2. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning
  3. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  4. 2024Cross-Domain Policy Transfer by Representation Alignment via Multi-Domain Behavioral Cloning
  5. 2024Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation
  6. 2024Cross Domain Policy Transfer with Effect Cycle-Consistency
  7. 2025In-Context Policy Adaptation via Cross-Domain Skill Diffusion
  8. 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning