# Learning Discrete State Abstractions With Deep Variational Inference

arXiv: https://arxiv.org/abs/2003.04300 · HTML: https://mlsanity.com/paper/2003.04300/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2020.

## Built on by (most cited first, 10 of 10)

- [Learning Markov State Abstractions for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04379/) (2021)
- [Proper Value Equivalence](https://mlsanity.com/paper/2106.10316/) (2021)
- [Continuous MDP Homomorphisms and Homomorphic Policy Gradient](https://mlsanity.com/paper/2209.07364/) (2022)
- [Learning Representations for Pixel-based Control: What Matters and Why?](https://mlsanity.com/paper/2111.07775/) (2021)
- [Policy Gradient Methods in the Presence of Symmetries and State Abstractions](https://mlsanity.com/paper/2305.05666/) (2023)
- [Computing Complexity-aware Plans Using Kolmogorov Complexity](https://mlsanity.com/paper/2109.10303/) (2021)
- [Discrete State-Action Abstraction via the Successor Representation](https://mlsanity.com/paper/2206.03467/) (2022)
- [Low-Variance Policy Gradient Estimation with World Models](https://mlsanity.com/paper/2010.15622/) (2020)
- [Using Forwards-Backwards Models to Approximate MDP Homomorphisms](https://mlsanity.com/paper/2209.06356/) (2022)
- [Adaptive state-action abstractions via rate-distortion](https://mlsanity.com/paper/2606.06123/) (2026)

## Abstract (arXiv)

Abstraction is crucial for effective sequential decision making in domains with large state spaces. In this work, we propose an information bottleneck method for learning approximate bisimulations, a type of state abstraction. We use a deep neural encoder to map states onto continuous embeddings. We map these embeddings onto a discrete representation using an action-conditioned hidden Markov model, which is trained end-to-end with the neural network. Our method is suited for environments with high-dimensional states and learns from a stream of experience collected by an agent acting in a Markov decision process. Through this learned discrete abstract model, we can efficiently plan for unseen goals in a multi-goal Reinforcement Learning setting. We test our method in simplified robotic manipulation domains with image states. We also compare it against previous model-based approaches to finding bisimulations in discrete grid-world-like environments. Source code is available at https://github.com/ondrejba/discrete_abstractions.