mlsanityData to 14 Oct 2026

ODRL: A Benchmark for Off-Dynamics Reinforcement Learning

arXiv 2410.20750 ↗First posted 28 Oct 2024cs.LG cs.AIcited by 25

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

We consider off-dynamics reinforcement learning (RL) where one needs to transfer policies across different domains with dynamics mismatch. Despite the focus on developing dynamics-aware algorithms, this field is hindered due to the lack of a standard benchmark. To bridge this gap, we introduce ODRL, the first benchmark tailored for evaluating off-dynamics RL methods. ODRL contains four experimental settings where the source and target domains can be either online or offline, and provides diverse tasks and a broad spectrum of dynamics shifts, making it a reliable platform to comprehensively evaluate the agent's adaptation ability to the target domain. Furthermore, ODRL includes recent off-dynamics RL algorithms in a unified framework and introduces some extra baselines for different settings, all implemented in a single-file manner. To unpack the true adaptation capability of existing methods, we conduct extensive benchmarking experiments, which show that no method has universal advantages across varied dynamics shifts. We hope this benchmark can serve as a cornerstone for future research endeavors. Our code is publicly available at https://github.com/OffDynamicsRL/off-dynamics-rl.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 56 of its 87 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

56 of 87 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatch24 May
  2. 2024Policy Learning for Off-Dynamics RL with Deficient Support16 Feb
  3. 2024Exploration and Anti-Exploration with Distributional Random Network Distillation18 Jan

Other references · 53

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  2. 2016OpenAI Gymcited by 1,736
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  7. 2018DeepMind Control Suitecited by 956
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  12. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
38 more
  1. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  2. 2019Active Domain Randomizationcited by 170
  3. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  4. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 141
  5. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
  6. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
  7. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  8. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 97
  9. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 85
  10. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  11. 2018Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translationcited by 50
  12. 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49
  13. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 46
  14. 2021Cross-domain Imitation from Observationscited by 42
  15. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
  16. 2019Environment Probing Interaction Policiescited by 39
  17. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  18. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  19. 2022i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loopscited by 32
  20. 2020Hierarchically Decoupled Imitation for Morphological Transfercited by 31
  21. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 31
  22. 2019How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?cited by 25
  23. 2023Universal Morphology Control via Contextual Modulationcited by 23
  24. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  25. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imaginationcited by 21
  26. 2019MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamicscited by 18
  27. 2022Robust Policy Learning over Multiple Uncertainty Setscited by 18
  28. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  29. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 17
  30. 2017Benchmark Environments for Multitask Learning in Continuous Domainscited by 14
  31. 2019Robust Visual Domain Randomization for Reinforcement Learningcited by 13
  32. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
  33. 2021Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Groundingcited by 12
  34. 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learningcited by 12
  35. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  36. 2019Domain Adaptive Imitation Learningcited by 6
  37. 2022State Advantage Weighting for Offline RLcited by 5
  38. 2020Traversing the Reality Gap via Simulator Tuningcited by 4

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 14.

25 papers.

Sort:

By area:

  1. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  2. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  3. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Datacited by 9
  4. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 9
  5. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  6. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 6
  7. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  8. 2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimationcited by 2
  9. 2025ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rolloutscited by 2
  10. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filteringcited by 2
  11. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  12. 2025Balance Equation-based Distributionally Robust Offline Imitation Learningcited by 1
13 more
  1. 2026DADP: Domain Adaptive Diffusion Policycited by 1
  2. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correctioncited by 1
  3. 2026Semi-Supervised Cross-Domain Imitation Learningcited by 1
  4. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learningcited by 1
  5. 2025Target-Aligned Fusion for Decision-Sequence Learning under Dynamics Shiftcited by 0
  6. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  7. 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learningcited by 0
  8. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0
  9. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learningcited by 0
  10. 2026DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learningcited by 0
  11. 2026Diagnose, Recover, Certify: Task Readiness under Hidden Dynamics Changescited by 0
  12. 2026When Does Test-Time Physical Diagnosis Pay? A Frozen Policy Buys Evidence It Never Readscited by 0
  13. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  2. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  3. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  4. 2025A Clean Slate for Offline Reinforcement Learning
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  6. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  7. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  8. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning