mlsanityData to 7 Oct 2026

MICo: Improved representations via sampling-based state similarity for Markov decision processes

arXiv 2106.08229 ↗First posted 3 Jun 2021cs.LG cs.AIcited by 43

Not on the map. Why?

Abstract (arXiv)

We present a new behavioural distance over the state space of a Markov decision process, and demonstrate the use of this distance as an effective means of shaping the learnt representations of deep reinforcement learning agents. While existing notions of state similarity are typically difficult to learn at scale due to high computational cost and lack of sample-based algorithms, our newly-proposed distance addresses both of these issues. In addition to providing detailed theoretical analysis, we provide empirical evidence that learning this distance alongside the value function yields structured and informative representations, including strong results on the Arcade Learning Environment benchmark.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 29 of its 81 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

29 of 81 references.

References · 29

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  3. 2018DeepMind Control Suitecited by 935
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  5. 2018Computational Optimal Transportcited by 765
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  8. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  9. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  10. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  11. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  12. 2017Learning Generative Models with Sinkhorn Divergencescited by 310
17 more
  1. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  2. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  3. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  4. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  5. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  6. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  7. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  8. 2019Learning with minibatch Wasserstein : asymptotic and gradient propertiescited by 83
  9. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  10. 2020Munchausen Reinforcement Learningcited by 68
  11. 2021Minibatch optimal transport distances; analysis and applicationscited by 62
  12. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 58
  13. 2018Combined Reinforcement Learning via Abstract Representationscited by 37
  14. 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
  15. 2021Metrics and continuity in reinforcement learningcited by 26
  16. 2020Efficient Wasserstein Natural Gradients for Reinforcement Learningcited by 23
  17. 2019ExTra: Transfer-guided Explorationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 6; 2024: 9; 2025: 9; 2026: 12.

43 papers.

Sort:

By area:

  1. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  2. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  3. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  4. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  5. 2022Continuous MDP Homomorphisms and Homomorphic Policy Gradientcited by 24
  6. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  7. 2026Predictive but Not Plannable: RC-aux for Latent World Modelscited by 16
  8. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
  9. 2025Horizon Generalization in Reinforcement Learningcited by 15
  10. 2021SimSR: Simple Distance-based State Representation for Deep Reinforcement Learningcited by 14
  11. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12
  12. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
31 more
  1. 2022Approximate Policy Iteration with Bisimulation Metricscited by 11
  2. 2023Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learningcited by 10
  3. 2024Agent-state based policies in POMDPs: Beyond belief-state MDPscited by 9
  4. 2023Policy Gradient Methods in the Presence of Symmetries and State Abstractionscited by 8
  5. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 7
  6. 2024The Curse of Diversity in Ensemble-Based Explorationcited by 6
  7. 2024Periodic agent-state based Q-learning for POMDPscited by 6
  8. 2024Learning Action-based Representations Using Invariancecited by 5
  9. 2024CALE: Continuous Arcade Learning Environmentcited by 5
  10. 2022Learning Generalizable Representations for Reinforcement Learning via Adaptive Meta-learner of Behavioral Similaritiescited by 4
  11. 2025Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learningcited by 4
  12. 2026BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agentscited by 4
  13. 2023State-Action Similarity-Based Representations for Off-Policy Evaluationcited by 3
  14. 2024An Empirical Study on the Power of Future Prediction in Partially Observable Environmentscited by 3
  15. 2024State Chrono Representation for Enhancing Generalization in Reinforcement Learningcited by 3
  16. 2025Reward-Aware Proto-Representations in Reinforcement Learningcited by 3
  17. 2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
  18. 2026Latent State Design for World Models under Sufficiency Constraintscited by 3
  19. 2026MSPR: Multi-scale Predictive Representations for Goal-conditioned Reinforcement Learningcited by 1
  20. 2026Task-Induced Representational Invariances Depend on Learning Objective in Deep RLcited by 1
  21. 2022Structural Similarity for Improved Transfer in Reinforcement Learningcited by 0
  22. 2023Wasserstein Auto-encoded MDPs: Formal Verification of Efficiently Distilled RL Policies with Many-sided Guaranteescited by 0
  23. 2023Towards Control-Centric Representations in Reinforcement Learning from Imagescited by 0
  24. 2025Approximated Behavioral Metric-based State Projection for Federated Reinforcement Learningcited by 0
  25. 2026SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Controlcited by 0
  26. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
  27. 2026Adaptive state-action abstractions via rate-distortioncited by 0
  28. 2026Cheap Reward Hacking Detectioncited by 0
  29. 2026BIFROST: Bridging Invariant Feature Representation for Observation-space Sim2Real Transfercited by 0
  30. 2026Sub-Quadratic Bisimulation Metrics via Approximate Nearest Neighbors: Coverage-Augmented Guarantees and Computable Two-Sided Certificatescited by 0
  31. 2026Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?cited by 0