# Online Abstraction with MDP Homomorphisms for Deep Learning

arXiv: https://arxiv.org/abs/1811.12929 · HTML: https://mlsanity.com/paper/1811.12929/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2018.

## Built on by (most cited first, 13 of 13)

- [Towards Robust Bisimulation Metric Learning](https://mlsanity.com/paper/2110.14096/) (2021)
- [Return-Based Contrastive Representation Learning for Reinforcement Learning](https://mlsanity.com/paper/2102.10960/) (2021)
- [Continuous MDP Homomorphisms and Homomorphic Policy Gradient](https://mlsanity.com/paper/2209.07364/) (2022)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Policy Gradient Methods in the Presence of Symmetries and State Abstractions](https://mlsanity.com/paper/2305.05666/) (2023)
- [Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstraction](https://mlsanity.com/paper/2407.13364/) (2024)
- [Fairness and Sequential Decision Making: Limits, Lessons, and Opportunities](https://mlsanity.com/paper/2301.05753/) (2023)
- [An Analysis of Model-Based Reinforcement Learning From Abstracted Observations](https://mlsanity.com/paper/2208.14407/) (2022)
- [Discovering User Types: Mapping User Traits by Task-Specific Behaviors in Reinforcement Learning](https://mlsanity.com/paper/2307.08169/) (2023)
- [Task-Induced Representational Invariances Depend on Learning Objective in Deep RL](https://mlsanity.com/paper/2606.01868/) (2026)
- [More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences](https://mlsanity.com/paper/2110.10632/) (2021)
- [Using Forwards-Backwards Models to Approximate MDP Homomorphisms](https://mlsanity.com/paper/2209.06356/) (2022)
- [Verification-Guided Falsification for Safe RL via Explainable Abstraction and Risk-Aware Exploration](https://mlsanity.com/paper/2506.03469/) (2025)

## Abstract (arXiv)

Abstraction of Markov Decision Processes is a useful tool for solving complex problems, as it can ignore unimportant aspects of an environment, simplifying the process of learning an optimal policy. In this paper, we propose a new algorithm for finding abstract MDPs in environments with continuous state spaces. It is based on MDP homomorphisms, a structure-preserving mapping between MDPs. We demonstrate our algorithm's ability to learn abstractions from collected experience and show how to reuse the abstractions to guide exploration in new tasks the agent encounters. Our novel task transfer method outperforms baselines based on a deep Q-network in the majority of our experiments. The source code is at https://github.com/ondrejba/aamas_19.