mlsanityData to 7 Oct 2026

Continuous MDP Homomorphisms and Homomorphic Policy Gradient

arXiv 2209.07364 ↗First posted 15 Sep 2022cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Abstraction has been widely studied as a way to improve the efficiency and generalization of reinforcement learning algorithms. In this paper, we study abstraction in the continuous-control setting. We extend the definition of MDP homomorphisms to encompass continuous actions in continuous state spaces. We derive a policy gradient theorem on the abstract MDP, which allows us to leverage approximate symmetries of the environment for policy optimization. Based on this theorem, we propose an actor-critic algorithm that is able to learn the policy and the MDP homomorphism map simultaneously, using the lax bisimulation metric. We demonstrate the effectiveness of our method on benchmark tasks in the DeepMind Control Suite. Our method's ability to utilize MDP homomorphisms for representation learning leads to improved performance when learning from pixel observations.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 57 of its 106 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

57 of 106 references.

References · 57

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2018World Modelscited by 1,444
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  12. 2019Hamiltonian Neural Networkscited by 702
38 more
  1. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  2. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  3. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  4. 2020Lagrangian Neural Networkscited by 388
  5. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  6. 2018Towards a Definition of Disentangled Representationscited by 294
  7. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  8. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  9. 2018Distributed Prioritized Experience Replaycited by 260
  10. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  11. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  12. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  13. 2019Unsupervised State Representation Learning in Ataricited by 144
  14. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  15. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 133
  16. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  17. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  18. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  19. 2021APS: Active Pretraining with Successor Featurescited by 109
  20. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
  21. 2019Self-Supervised Generalisation with Meta Auxiliary Learningcited by 88
  22. 2020The Value Equivalence Principle for Model-Based Reinforcement Learningcited by 88
  23. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  24. 2022SO(2)-Equivariant Reinforcement Learningcited by 79
  25. 2019Learning Action Representations for Reinforcement Learningcited by 75
  26. 2017Near Optimal Behavior via Approximate State Abstractioncited by 63
  27. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  28. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  29. 2022Learning Symmetric Embeddings for Equivariant World Modelscited by 54
  30. 2021Towards Robust Bisimulation Metric Learningcited by 49
  31. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  32. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 42
  33. 2019Symmetry-Based Disentangled Representation Learning requires Interaction with Environmentscited by 35
  34. 2019Dynamics-aware Embeddingscited by 34
  35. 2021Proper Value Equivalencecited by 33
  36. 2019The Natural Language of Actionscited by 28
  37. 2021Multi-Agent MDP Homomorphic Networkscited by 24
  38. 2020Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarkingcited by 18

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 7; 2025: 6; 2026: 6.

24 papers.

Sort:

By area:

  1. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  2. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  3. 2023EDGI: Equivariant Diffusion for Planning with Embodied Agentscited by 35
  4. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
  5. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  6. 2024Morphological Symmetries in Roboticscited by 9
  7. 2024iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learningcited by 9
  8. 2023Policy Gradient Methods in the Presence of Symmetries and State Abstractionscited by 8
  9. 2024Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstractioncited by 6
  10. 2023rm E(3)-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learningcited by 4
  11. 2023Equivariant MuZerocited by 3
  12. 2024SiT: Symmetry-Invariant Transformers for Generalisation in Reinforcement Learningcited by 3
12 more
  1. 2024Leveraging Symmetry to Accelerate Learning of Trajectory Tracking Controllers for Free-Flying Robotic Systemscited by 2
  2. 2024Reinforcement Learning with Euclidean Data Augmentation for State-Based Continuous Controlcited by 2
  3. 2025Learning Abstract World Models with a Group-Structured Latent Spacecited by 1
  4. 2025Universal Reinforcement Learning in Coalgebras: Asynchronous Stochastic Computation via Conductioncited by 1
  5. 2026Task-Induced Representational Invariances Depend on Learning Objective in Deep RLcited by 1
  6. 2025Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processescited by 0
  7. 2025Homomorphic Mappings for Value-Preserving State Aggregation in Markov Decision Processescited by 0
  8. 2026Delayed homomorphic reinforcement learning for environments with delayed feedbackcited by 0
  9. 2026TERRA: Task-Embedded Reasoning and Representation Architecture for Cross-Domain Applicationscited by 0
  10. 2026Adaptive state-action abstractions via rate-distortioncited by 0
  11. 2026Compositional Behavioral Semantics for State Abstraction in Reinforcement Learningcited by 0
  12. 2026BIFROST: Bridging Invariant Feature Representation for Observation-space Sim2Real Transfercited by 0