mlsanityData to 7 Oct 2026

Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning

arXiv 2110.12997 ↗First posted 25 Oct 2021cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Unsupervised reinforcement learning aims to acquire skills without prior goal representations, where an agent automatically explores an open-ended environment to represent goals and learn the goal-conditioned policy. However, this procedure is often time-consuming, limiting the rollout in some potentially expensive target environments. The intuitive approach of training in another interaction-rich environment disrupts the reproducibility of trained skills in the target environment due to the dynamics shifts and thus inhibits direct transferring. Assuming free access to a source environment, we propose an unsupervised domain adaptation method to identify and acquire skills across dynamics. Particularly, we introduce a KL regularized objective to encourage emergence of skills, rewarding the agent for both discovering skills and aligning its behaviors respecting dynamics shifts. This suggests that both dynamics (source and target) shape the reward to facilitate the learning of adaptive skills. We also conduct empirical experiments to demonstrate that our method can effectively learn skills that can be smoothly deployed in target.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 45 of its 61 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

45 of 61 references.

References · 45

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  5. 2016OpenAI Gymcited by 1,684
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  10. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  11. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  12. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
33 more
  1. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  2. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  3. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  4. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  5. 2019Efficient Exploration via State Marginal Matchingcited by 186
  6. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  7. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  8. 2019Unsupervised State Representation Learning in Ataricited by 144
  9. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  10. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  11. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  12. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  13. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  14. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  15. 2019State Alignment-based Imitation Learningcited by 73
  16. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  17. 2019Information asymmetry in KL-regularized RLcited by 53
  18. 2017Divide-and-Conquer Reinforcement Learningcited by 46
  19. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 41
  20. 2019Contextual Imagined Goals for Self-Supervised Robotic Learningcited by 40
  21. 2020State-only Imitation with Transition Dynamics Mismatchcited by 37
  22. 2019Unsupervised Curricula for Visual Meta-Reinforcement Learningcited by 33
  23. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  24. 2020Action and Perception as Divergence Minimizationcited by 31
  25. 2019Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitivescited by 30
  26. 2019TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfercited by 21
  27. 2019Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learningcited by 18
  28. 2017Mutual Alignment Transfer Learningcited by 16
  29. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 15
  30. 2018Learning to Share and Hide Intentions using Information Regularizationcited by 13
  31. 2019Disentangled Skill Embeddings for Reinforcement Learningcited by 9
  32. 2019SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environmentscited by 9
  33. 2020GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learningcited by 9

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 7; 2024: 5.

15 papers.

Sort:

By area:

  1. 2023Behavior Proximal Policy Optimizationcited by 43
  2. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  3. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  4. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  5. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  6. 2024Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitationcited by 15
  7. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  8. 2024Scalable and Effective Arithmetic Tree Generation for Adder and Multiplier Designscited by 9
  9. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  10. 2023SAM-OCTA: Prompting Segment-Anything for OCTA Image Segmentationcited by 5
  11. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  12. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 4
3 more
  1. 2022A Transferable Legged Mobile Manipulation Framework Based on Disturbance Predictive Controlcited by 1
  2. 2022KSG: Knowledge and Skill Graphcited by 1
  3. 2024Off-dynamics Conditional Diffusion Plannerscited by 0