mlsanityData to 7 Oct 2026

Learning Domain Invariant Representations in Goal-conditioned Block MDPs

arXiv 2110.14248 ↗First posted 27 Oct 2021cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Deep Reinforcement Learning (RL) is successful in solving many complex Markov Decision Processes (MDPs) problems. However, agents often face unanticipated environmental changes after deployment in the real world. These changes are often spurious and unrelated to the underlying problem, such as background shifts for visual input agents. Unfortunately, deep RL policies are usually sensitive to these changes and fail to act robustly against them. This resembles the problem of domain generalization in supervised learning. In this work, we study this problem for goal-conditioned RL agents. We propose a theoretical framework in the Block MDP setting that characterizes the generalizability of goal-conditioned policies to new environments. Under this framework, we develop a practical method PA-SkewFit that enhances domain generalization. The empirical evaluation shows that our goal-conditioned RL agent can perform well in various unseen test environments, improving by 50% over baselines.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

40 of 56 references.

References · 40

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2019Invariant Risk Minimizationcited by 1,832
  6. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2020Reinforcement Learning with Augmented Datacited by 452
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  12. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
28 more
  1. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  2. 2015The Variational Fair Autoencodercited by 223
  3. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  4. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  5. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  6. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  7. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  8. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  9. 2019Generalizing to unseen domains via distribution matchingcited by 109
  10. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  11. 2020Invariant Causal Prediction for Block MDPscited by 104
  12. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  13. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  14. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  15. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  16. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  17. 2019Exploration via Hindsight Goal Generationcited by 53
  18. 2021Domain Adversarial Neural Networks for Domain Generalization: When It Works and How to Improvecited by 46
  19. 2020Goal-Aware Prediction: Learning to Model What Matterscited by 45
  20. 2019Adversarial Invariant Feature Learning with Accuracy Constraint for Domain Generalizationcited by 43
  21. 2020Planning from Pixels using Inverse Dynamics Modelscited by 36
  22. 2020Feature Alignment and Restoration for Domain Generalization and Adaptationcited by 30
  23. 2018Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Explorationcited by 23
  24. 2020Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learningcited by 23
  25. 2014FastMMD: Ensemble of Circular Discrepancy for Efficient Two-Sample Testcited by 17
  26. 2020Weakly-Supervised Reinforcement Learning for Controllable Behaviorcited by 17
  27. 2020ROLL: Visual Self-Supervised Reinforcement Learning with Object Reasoningcited by 10
  28. 2020A Geometric Perspective on Self-Supervised Policy Adaptationcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 5; 2024: 3.

12 papers.

  1. 2023A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidancecited by 40
  2. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  3. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  4. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  5. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  6. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  7. 2022The Two Dimensions of Worst-case Training and the Integrated Effect for Out-of-domain Generalizationcited by 21
  8. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
  9. 2024Entity-based Reinforcement Learning for Autonomous Cyber Defencecited by 11
  10. 2024How to Solve Contextual Goal-Oriented Problems with Offline Datasets?cited by 1
  11. 2024The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Datacited by 1
  12. 2023Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalizationcited by 0