# Exploiting Multiple Abstractions in Episodic RL via Reward Shaping

arXiv: https://arxiv.org/abs/2303.00516 · HTML: https://mlsanity.com/paper/2303.00516/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 29 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 6 of 6)

- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Near Optimal Behavior via Approximate State Abstraction](https://mlsanity.com/paper/1701.04113/) (2017)
- [Abstract Value Iteration for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2010.15638/) (2020)
- [Plan-Based Relaxed Reward Shaping for Goal-Directed Tasks](https://mlsanity.com/paper/2107.06661/) (2021)
- [Hierarchical Representation Learning for Markov Decision Processes](https://mlsanity.com/paper/2106.01655/) (2021)

## Built on by (most cited first, 2 of 2)

- [On the Sample Efficiency of Abstractions and Potential-Based Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2404.07826/) (2024)
- [Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2512.04958/) (2025)

## Abstract (arXiv)

One major limitation to the applicability of Reinforcement Learning (RL) to many practical domains is the large number of samples required to learn an optimal policy. To address this problem and improve learning efficiency, we consider a linear hierarchy of abstraction layers of the Markov Decision Process (MDP) underlying the target domain. Each layer is an MDP representing a coarser model of the one immediately below in the hierarchy. In this work, we propose a novel form of Reward Shaping where the solution obtained at the abstract level is used to offer rewards to the more concrete MDP, in such a way that the abstract solution guides the learning in the more complex domain. In contrast with other works in Hierarchical RL, our technique has few requirements in the design of the abstract models and it is also tolerant to modeling errors, thus making the proposed approach practical. We formally analyze the relationship between the abstract models and the exploration heuristic induced in the lower-level domain. Moreover, we prove that the method guarantees optimal convergence and we demonstrate its effectiveness experimentally.