# Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers

arXiv: https://arxiv.org/abs/2006.13916 · HTML: https://mlsanity.com/paper/2006.13916/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2020.

## Built on by (most cited first, 30 of 97)

- [When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning](https://mlsanity.com/paper/2206.13464/) (2022)
- [RobustNav: Towards Benchmarking Robustness in Embodied Navigation](https://mlsanity.com/paper/2106.04531/) (2021)
- [DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning](https://mlsanity.com/paper/2203.06662/) (2022)
- [Mismatched No More: Joint Model-Policy Optimization for Model-Based RL](https://mlsanity.com/paper/2110.02758/) (2021)
- [Lifelong Robotic Reinforcement Learning by Retaining Experiences](https://mlsanity.com/paper/2109.09180/) (2021)
- [Rank2Reward: Learning Shaped Reward Functions from Passive Video](https://mlsanity.com/paper/2404.14735/) (2024)
- [Cross-Domain Policy Adaptation via Value-Guided Data Filtering](https://mlsanity.com/paper/2305.17625/) (2023)
- [Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective](https://mlsanity.com/paper/2209.08466/) (2022)
- [Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning](https://mlsanity.com/paper/2306.12755/) (2023)
- [Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation](https://mlsanity.com/paper/2402.15399/) (2024)
- [Cross-Domain Policy Adaptation by Capturing Representation Mismatch](https://mlsanity.com/paper/2405.15369/) (2024)
- [ODRL: A Benchmark for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.20750/) (2024)
- [You Only Live Once: Single-Life Reinforcement Learning](https://mlsanity.com/paper/2210.08863/) (2022)
- [State Regularized Policy Optimization on Data with Dynamics Shift](https://mlsanity.com/paper/2306.03552/) (2023)
- [Transfer RL across Observation Feature Spaces via Model-Based Regularization](https://mlsanity.com/paper/2201.00248/) (2022)
- [Pretraining in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2211.03959/) (2022)
- [Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning](https://mlsanity.com/paper/2405.06192/) (2024)
- [H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps](https://mlsanity.com/paper/2309.12716/) (2023)
- [Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation](https://mlsanity.com/paper/2411.09891/) (2024)
- [Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning](https://mlsanity.com/paper/2502.02705/) (2025)
- [A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agents](https://mlsanity.com/paper/2402.04580/) (2024)
- [Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2409.20521/) (2024)
- [Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning](https://mlsanity.com/paper/2110.12997/) (2021)
- [CTRLS: Chain-of-Thought Reasoning via Latent State-Transition](https://mlsanity.com/paper/2507.08182/) (2025)
- [Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.23450/) (2024)
- [On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluation](https://mlsanity.com/paper/2402.14703/) (2024)
- [Policy Learning for Off-Dynamics RL with Deficient Support](https://mlsanity.com/paper/2402.10765/) (2024)
- [xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editing](https://mlsanity.com/paper/2409.08687/) (2024)
- [Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data](https://mlsanity.com/paper/2505.23062/) (2025)
- [MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning](https://mlsanity.com/paper/2506.08460/) (2025)

## Abstract (arXiv)

We propose a simple, practical, and intuitive approach for domain adaptation in reinforcement learning. Our approach stems from the idea that the agent's experience in the source domain should look similar to its experience in the target domain. Building off of a probabilistic view of RL, we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function. This modified reward function is simple to estimate by learning auxiliary classifiers that distinguish source-domain transitions from target-domain transitions. Intuitively, the modified reward function penalizes the agent for visiting states and taking actions in the source domain which are not possible in the target domain. Said another way, the agent is penalized for transitions that would indicate that the agent is interacting with the source domain, rather than the target domain. Our approach is applicable to domains with continuous states and actions and does not require learning an explicit model of the dynamics. On discrete and continuous control tasks, we illustrate the mechanics of our approach and demonstrate its scalability to high-dimensional tasks.