mlsanityData to 14 Oct 2026

Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning

arXiv 2310.17139 ↗First posted 26 Oct 2023cs.LGcited by 10

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

While bisimulation-based approaches hold promise for learning robust state representations for Reinforcement Learning (RL) tasks, their efficacy in offline RL tasks has not been up to par. In some instances, their performance has even significantly underperformed alternative methods. We aim to understand why bisimulation methods succeed in online settings, but falter in offline tasks. Our analysis reveals that missing transitions in the dataset are particularly harmful to the bisimulation principle, leading to ineffective estimation. We also shed light on the critical role of reward scaling in bounding the scale of bisimulation measurements and of the value error they induce. Based on these findings, we propose to apply the expectile operator for representation learning to our offline RL setting, which helps to prevent overfitting to incomplete data. Meanwhile, by introducing an appropriate reward scaling strategy, we avoid the risk of feature collapse in representation space. We implement these recommendations on two state-of-the-art bisimulation-based algorithms, MICo and SimSR, and demonstrate performance gains on two benchmark suites: D4RL and Visual D4RL. Codes are provided at urlhttps://github.com/zanghyu/OfflineBisimulation.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 56 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

37 of 56 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting22 Jun
  2. 2022Behavior Prior Representation learning for Offline Reinforcement Learning2 Nov
  3. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  4. 2022Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learning2 Jun
  5. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Representation Matters: Offline Pretraining for Sequential Decision Making11 Feb
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 27

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  7. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  8. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  9. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  10. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  11. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  12. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
15 more
  1. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  2. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  3. 2020Deep Reinforcement and InfoMax Learningcited by 79
  4. 2021Towards Robust Bisimulation Metric Learningcited by 49
  5. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  6. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
  7. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  8. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  9. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 32
  10. 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
  11. 2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
  12. 2022An Empirical Investigation of Representation Learning for Imitationcited by 15
  13. 2021SimSR: Simple Distance-based State Representation for Deep Reinforcement Learningcited by 14
  14. 2020Learning Robust State Abstractions for Hidden-Parameter Block MDPscited by 4
  15. 2019ExTra: Transfer-guided Explorationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 4.

10 papers.

  1. 2025One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learningcited by 8
  2. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  3. 2025A Generalized Bisimulation Metric of State Similarity between Markov Decision Processes: From Theoretical Propositions to Applicationscited by 4
  4. 2025Learning Fused State Representations for Control from Multi-View Observationscited by 1
  5. 2026Task-Aware Exploration via a Predictive Bisimulation Metriccited by 1
  6. 2024Intrinsic Dynamics-Driven Generalizable Scene Representations for Vision-Oriented Decision-Making Applicationscited by 0
  7. 2024Stable Offline Value Function Learning with Bisimulation-based Representationscited by 0
  8. 2025A Theoretical Analysis of State Similarity Between Markov Decision Processescited by 0
  9. 2026On the Structural Non-Preservation of Epistemic Behaviour under Policy Transformationcited by 0
  10. 2026Adaptive state-action abstractions via rate-distortioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learning
  2. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learning
  3. 2026PAMD: Structured Adaptive Distances for Bisimulation Representations in Visual Reinforcement Learning
  4. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  5. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  6. 2024Simple Ingredients for Offline Reinforcement Learning
  7. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learning
  8. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces