Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Bisimulation metric has long been regarded as an effective control-related representation learning technique in various reinforcement learning tasks. However, in this paper, we identify two main issues with the conventional bisimulation metric: 1) an inability to represent certain distinctive scenarios, and 2) a reliance on predefined weights for differences in rewards and subsequent states during recursive updates. We find that the first issue arises from an imprecise definition of the reward gap, whereas the second issue stems from overlooking the varying importance of reward difference and next-state distinctions across different training stages and task settings. To address these issues, by introducing a measure for state-action pairs, we propose a revised bisimulation metric that features a more precise definition of reward gap and novel update operators with adaptive coefficient. We also offer theoretical guarantees of convergence for our proposed metric and its improved representation distinctiveness. In addition to our rigorous theoretical analysis, we conduct extensive experiments on two representative benchmarks, DeepMind Control and Meta-World, demonstrating the effectiveness of our approach.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 24 Jul 2025.
Builds on
22 of 32 references.
References · 22
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2023Mastering Diverse Domains through World Modelscited by 1,193
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
- 2020Mastering Atari with Discrete World Modelscited by 990
- 2018DeepMind Control Suitecited by 935
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
- 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
- 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
- 2022Temporal Difference Learning for Model Predictive Controlcited by 375
- 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
- 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
- 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
10 more
- 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
- 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
- 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
- 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
- 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
- 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
- 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
- 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
- 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
- 2021SimSR: Simple Distance-based State Representation for Deep Reinforcement Learningcited by 14
Built on by
Citing papers per year: 2026: 3.
3 papers.