mlsanityData to 7 Oct 2026

Cross-Domain Policy Adaptation via Value-Guided Data Filtering

arXiv 2305.17625 ↗First posted 28 May 2023cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Generalizing policies across different domains with dynamics mismatch poses a significant challenge in reinforcement learning. For example, a robot learns the policy in a simulator, but when it is deployed in the real world, the dynamics of the environment may be different. Given the source and target domain with dynamics mismatch, we consider the online dynamics adaptation problem, in which case the agent can access sufficient source domain data while online interactions with the target domain are limited. Existing research has attempted to solve the problem from the dynamics discrepancy perspective. In this work, we reveal the limitations of these methods and explore the problem from the value difference perspective via a novel insight on the value consistency across domains. Specifically, we present the Value-Guided Data Filtering (VGDF) algorithm, which selectively shares transitions from the source domain based on the proximity of paired value targets across the two domains. Empirical results on various environments with kinematic and morphology shifts demonstrate that our method achieves superior performance compared to prior approaches.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 60 of its 83 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

60 of 83 references.

References · 60

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
38 more
  1. 2019Solving Rubik's Cube with a Robot Handcited by 699
  2. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  5. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  6. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  7. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  8. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  9. 2018Selective Experience Replay for Lifelong Learningcited by 239
  10. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  11. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  12. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  13. 2018Generative Adversarial Imitation from Observationcited by 183
  14. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  15. 2019Active Domain Randomizationcited by 161
  16. 2019A review of domain adaptation without target labelscited by 154
  17. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  18. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  19. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 129
  20. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  21. 2020Conservative Safety Critics for Explorationcited by 107
  22. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  23. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  24. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  25. 2022Neural-Fly Enables Rapid Learning for Agile Flight in Strong Windscited by 80
  26. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 78
  27. 2019State Alignment-based Imitation Learningcited by 73
  28. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  29. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  30. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  31. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
  32. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  33. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  34. 2021Auto-Tuned Sim-to-Real Transfercited by 45
  35. 2020Data-efficient Domain Randomization with Bayesian Optimizationcited by 43
  36. 2020Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistencycited by 43
  37. 2018Policy Transfer with Strategy Optimizationcited by 41
  38. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 10; 2026: 10.

30 papers.

Sort:

By area:

  1. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  2. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
  3. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  4. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
  5. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15
  6. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  7. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  8. 2024Policy Learning for Off-Dynamics RL with Deficient Supportcited by 8
  9. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Datacited by 8
  10. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 8
  11. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  12. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
18 more
  1. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  2. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  3. 2024Offline Reinforcement Learning with Domain-Unlabeled Datacited by 3
  4. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  5. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filteringcited by 2
  6. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learningcited by 1
  7. 2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimationcited by 1
  8. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correctioncited by 1
  9. 2026Semi-Supervised Cross-Domain Imitation Learningcited by 1
  10. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learningcited by 1
  11. 2025Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learningcited by 0
  12. 2026Bridging Dynamics Gaps via Diffusion Schr"odinger Bridge for Cross-Domain Reinforcement Learningcited by 0
  13. 2026Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Criticscited by 0
  14. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  15. 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learningcited by 0
  16. 2026Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learningcited by 0
  17. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0
  18. 2026DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learningcited by 0