# Combined Reinforcement Learning via Abstract Representations

arXiv: https://arxiv.org/abs/1809.04506 · HTML: https://mlsanity.com/paper/1809.04506/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2018.

## Built on by (most cited first, 30 of 37)

- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Bridging State and History Representations: Understanding Self-Predictive RL](https://mlsanity.com/paper/2401.08898/) (2024)
- [Towards Robust Bisimulation Metric Learning](https://mlsanity.com/paper/2110.14096/) (2021)
- [MICo: Improved representations via sampling-based state similarity for Markov decision processes](https://mlsanity.com/paper/2106.08229/) (2021)
- [Return-Based Contrastive Representation Learning for Reinforcement Learning](https://mlsanity.com/paper/2102.10960/) (2021)
- [Proper Value Equivalence](https://mlsanity.com/paper/2106.10316/) (2021)
- [Factored Adaptation for Non-Stationary Reinforcement Learning](https://mlsanity.com/paper/2203.16582/) (2022)
- [Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning](https://mlsanity.com/paper/2207.14800/) (2022)
- [Investigating the Properties of Neural Network Representations in Reinforcement Learning](https://mlsanity.com/paper/2203.15955/) (2022)
- [Learning First-Order Representations for Planning from Black-Box States: New Results](https://mlsanity.com/paper/2105.10830/) (2021)
- [Domain Adversarial Reinforcement Learning](https://mlsanity.com/paper/2102.07097/) (2021)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Interpretable Machine Learning in Physics: A Review](https://mlsanity.com/paper/2503.23616/) (2025)
- [Reasoning-Modulated Representations](https://mlsanity.com/paper/2107.08881/) (2021)
- [Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloning](https://mlsanity.com/paper/2506.10137/) (2025)
- [Combining Planning and Learning of Behavior Trees for Robotic Assembly](https://mlsanity.com/paper/2103.09036/) (2021)
- [Low Dimensional State Representation Learning with Robotics Priors in Continuous Action Spaces](https://mlsanity.com/paper/2107.01667/) (2021)
- [Predictive auxiliary objectives in deep RL mimic learning in the brain](https://mlsanity.com/paper/2310.06089/) (2023)
- [Policy Gradients Incorporating the Future](https://mlsanity.com/paper/2108.02096/) (2021)
- [Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2210.07636/) (2022)
- [Learning Team-Based Navigation: A Review of Deep Reinforcement Learning Techniques for Multi-Agent Pathfinding](https://mlsanity.com/paper/2308.05893/) (2023)
- [Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization](https://mlsanity.com/paper/2112.06625/) (2021)
- [Off-policy Evaluation with Deeply-abstracted States](https://mlsanity.com/paper/2406.19531/) (2024)
- [Mutual Information Tracks Policy Coherence in Reinforcement Learning](https://mlsanity.com/paper/2509.10423/) (2025)
- [Learning List-wise Representation in Reinforcement Learning for Ads Allocation with Multiple Auxiliary Tasks](https://mlsanity.com/paper/2204.00888/) (2022)
- [Deep SPI: Safe Policy Improvement via World Models](https://mlsanity.com/paper/2510.12312/) (2025)
- [Low-Dimensional State and Action Representation Learning with MDP Homomorphism Metrics](https://mlsanity.com/paper/2107.01677/) (2021)
- [Information-Bottleneck-Based Behavior Representation Learning for Multi-agent Reinforcement learning](https://mlsanity.com/paper/2109.14188/) (2021)
- [Component Transfer Learning for Deep RL Based on Abstract Representations](https://mlsanity.com/paper/2111.11525/) (2021)

## Abstract (arXiv)

In the quest for efficient and robust reinforcement learning methods, both model-free and model-based approaches offer advantages. In this paper we propose a new way of explicitly bridging both approaches via a shared low-dimensional learned encoding of the environment, meant to capture summarizing abstractions. We show that the modularity brought by this approach leads to good generalization while being computationally efficient, with planning happening in a smaller latent state space. In addition, this approach recovers a sufficient low-dimensional representation of the environment, which opens up new strategies for interpretable AI, exploration and transfer learning.