mlsanityData to 14 Oct 2026

Learning Action-based Representations Using Invariance

arXiv 2403.16369 ↗First posted 25 Mar 2024cs.LG cs.AI stat.MLcited by 7

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Robust reinforcement learning agents using high-dimensional observations must be able to identify relevant state features amidst many exogeneous distractors. A representation that captures controllability identifies these state elements by determining what affects agent control. While methods such as inverse dynamics and mutual information capture controllability for a limited number of timesteps, capturing long-horizon elements remains a challenging problem. Myopic controllability can capture the moment right before an agent crashes into a wall, but not the control-relevance of the wall while the agent is still some distance away. To address this we introduce action-bisimulation encoding, a method inspired by the bisimulation invariance pseudometric, that extends single-step controllability with a recursive invariance constraint. By doing this, action-bisimulation learns a multi-step controllability metric that smoothly discounts distant state features that are relevant for control. We demonstrate that action-bisimulation pretraining on reward-free, uniformly random data improves sample efficiency in several environments, including a photorealistic 3D simulation domain, Habitat. Additionally, we provide theoretical analysis and qualitative results demonstrating the information captured by action-bisimulation.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 60 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

34 of 60 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2018World Modelscited by 1,513
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  10. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  12. 2015Deep Kalman Filterscited by 166
22 more
  1. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 156
  2. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
  3. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  4. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  5. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  6. 2020Invariant Causal Prediction for Block MDPscited by 105
  7. 2017Near Optimal Behavior via Approximate State Abstractioncited by 66
  8. 2021Towards Robust Bisimulation Metric Learningcited by 49
  9. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 45
  10. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  11. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  12. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  13. 2020What can I do here? A Theory of Affordances in Reinforcement Learningcited by 35
  14. 2018Contingency-Aware Exploration in Reinforcement Learningcited by 26
  15. 2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
  16. 2023Robust Representation Learning by Clustering with Bisimulation Metrics for Visual Reinforcement Learning with Distractionscited by 14
  17. 2022Approximate Policy Iteration with Bisimulation Metricscited by 12
  18. 2013Model Reduction Techniques for Computing Approximately Optimal Solutions for Markov Decision Processescited by 10
  19. 2019Hypothesis-Driven Skill Discovery for Hierarchical Deep Reinforcement Learningcited by 8
  20. 2023Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learningcited by 7
  21. 2022Efficient Embedding of Semantic Similarity in Control Policies via Entangled Bisimulationcited by 3
  22. 2020Disentangling Controllable Object through Video Prediction Improves Visual Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  2. 2025From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learningcited by 3
  3. 2026Predictive Objectives Discard Exogenous Control-Relevant Features: A Controlled Mechanistic Studycited by 2
  4. 2026Action-Sufficient Goal Representationscited by 1
  5. 2026SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Controlcited by 0
  6. 2026Exploiting Local Dynamics Regularity for Reusable Skills in Offline Hierarchical RLcited by 0
  7. 2026Learning Object Manipulation from Scratch via Contrastive Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learning
  2. 2024Learning Latent Dynamic Robust Representations for World Models
  3. 2025Learning Fused State Representations for Control from Multi-View Observations
  4. 2023Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning
  5. 2026Factorized Spectral Representations for Reinforcement Learning
  6. 2026Bilinear World Models: Learning Representations with Structured Dynamics for Efficient Control
  7. 2026PAMD: Structured Adaptive Distances for Bisimulation Representations in Visual Reinforcement Learning
  8. 2026BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents