mlsanityData to 7 Oct 2026

Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning

arXiv 2204.13060 ↗First posted 27 Apr 2022cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Building generalizable goal-conditioned agents from rich observations is a key to reinforcement learning (RL) solving real world problems. Traditionally in goal-conditioned RL, an agent is provided with the exact goal they intend to reach. However, it is often not realistic to know the configuration of the goal before performing a task. A more scalable framework would allow us to provide the agent with an example of an analogous task, and have the agent then infer what the goal should be for its current state. We propose a new form of state abstraction called goal-conditioned bisimulation that captures functional equivariance, allowing for the reuse of skills to achieve new goals. We learn this representation using a metric form of this abstraction, and show its ability to generalize to new goals in simulation manipulation tasks. Further, we prove that this learned representation is sufficient not only for goal conditioned tasks, but is amenable to any downstream task described by a state-only reward function. Videos can be found at https://sites.google.com/view/gc-bisimulation.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 27 Apr 2022.

Builds on

37 of 59 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  9. 2020Reinforcement Learning with Augmented Datacited by 452
  10. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  12. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
25 more
  1. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  2. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  3. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  4. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  5. 2019Unsupervised State Representation Learning in Ataricited by 144
  6. 2019Planning with Goal-Conditioned Policiescited by 144
  7. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  8. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  9. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  10. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  11. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  12. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  13. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 53
  14. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  15. 2015Learning image representations tied to ego-motioncited by 41
  16. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 41
  17. 2019Contextual Imagined Goals for Self-Supervised Robotic Learningcited by 40
  18. 2019Learning Robotic Manipulation through Visual Planning and Actingcited by 39
  19. 2018Natural Environment Benchmarks for Reinforcement Learningcited by 32
  20. 2018Grasp2Vec: Learning Object Representations from Self-Supervised Graspingcited by 27
  21. 2020Hallucinative Topological Memory for Zero-Shot Visual Planningcited by 25
  22. 2021What Can I Do Here? Learning New Skills by Imagining Visual Affordancescited by 22
  23. 2020Plan2Vec: Unsupervised Representation Learning by Latent Planscited by 21
  24. 2017PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representationscited by 16
  25. 2021Learning Domain Invariant Representations in Goal-conditioned Block MDPscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 8; 2025: 10; 2026: 10.

38 papers.

Sort:

By area:

  1. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  2. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  3. 2023Contrastive Difference Predictive Codingcited by 35
  4. 2022Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptationcited by 28
  5. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  6. 2023Learning to Extrapolate: A Transductive Approachcited by 19
  7. 2025Horizon Generalization in Reinforcement Learningcited by 15
  8. 2023Memory-Consistent Neural Networks for Imitation Learningcited by 14
  9. 2023A Survey of Demonstration Learningcited by 12
  10. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weightingcited by 12
  11. 2023Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learningcited by 10
  12. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
26 more
  1. 2023Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangementcited by 6
  2. 2024Generalize by Touching: Tactile Ensemble Skill Transfer for Robotic Furniture Assemblycited by 6
  3. 2024Bisimulation Metrics are Optimal Transport Distances, and Can be Computed Efficientlycited by 6
  4. 2024Learning Action-based Representations Using Invariancecited by 5
  5. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
  6. 2025A Generalized Bisimulation Metric of State Similarity between Markov Decision Processes: From Theoretical Propositions to Applicationscited by 4
  7. 2023A Kernel Perspective on Behavioural Metrics for Markov Decision Processescited by 3
  8. 2024State Chrono Representation for Enhancing Generalization in Reinforcement Learningcited by 3
  9. 2025Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Taskscited by 3
  10. 2025MultiModal Action Conditioned Video Generationcited by 3
  11. 2024Fairness in Reinforcement Learning with Bisimulation Metricscited by 1
  12. 2025Unifying Agent Interaction and World Information for Multi-agent Coordinationcited by 1
  13. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1
  14. 2026Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RLcited by 1
  15. 2023Learning Embeddings for Sequential Tasks Using Population of Agentscited by 0
  16. 2025Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learningcited by 0
  17. 2025Learning Causal States Under Partial Observability and Perturbationcited by 0
  18. 2025A Theoretical Analysis of State Similarity Between Markov Decision Processescited by 0
  19. 2026Intrinsic Vicarious Conditioning for Deep Reinforcement Learningcited by 0
  20. 2026Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RLcited by 0
  21. 2026Learning to Perceive the World Through Control: Empowerment-Based Representation Learningcited by 0
  22. 2026Neuro-Symbolic Learning for Long-Horizon Task Planning Under Complex Logical Constraintscited by 0
  23. 2026NeSy-CSA: A Neuro-Symbolic Framework for Open-Ended Critical Scenario Attributioncited by 0
  24. 2026What Is Worth Representing? Representational Empowerment for Continual Model Constructioncited by 0
  25. 2026VERA: Verifiable Feasibility Representations with Counterfactual Credit for Constrained Multi-Agent Controlcited by 0
  26. 2026Learning Transferable Skills using Goal-Conditioned Bisimulationcited by 0