# Planning by Prioritized Sweeping with Small Backups

arXiv: https://arxiv.org/abs/1301.2343 · HTML: https://mlsanity.com/paper/1301.2343/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2013.

## Built on by (most cited first, 6 of 6)

- [Regret Minimization Experience Replay in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2105.07253/) (2021)
- [MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimization](https://mlsanity.com/paper/2302.10418/) (2023)
- [Characterizing the Gap Between Actor-Critic and Policy Gradient](https://mlsanity.com/paper/2106.06932/) (2021)
- [Doubly-Asynchronous Value Iteration: Making Value Iteration Asynchronous in Actions](https://mlsanity.com/paper/2207.01613/) (2022)
- [Attention Loss Adjusted Prioritized Experience Replay](https://mlsanity.com/paper/2309.06684/) (2023)
- [Spectral Prioritized Sweeping in Nonstationary Reinforcement Learning](https://mlsanity.com/paper/2609.06186/) (2026)

## Abstract (arXiv)

Efficient planning plays a crucial role in model-based reinforcement learning. Traditionally, the main planning operation is a full backup based on the current estimates of the successor states. Consequently, its computation time is proportional to the number of successor states. In this paper, we introduce a new planning backup that uses only the current value of a single successor state and has a computation time independent of the number of successor states. This new backup, which we call a small backup, opens the door to a new class of model-based reinforcement learning methods that exhibit much finer control over their planning process than traditional methods. We empirically demonstrate that this increased flexibility allows for more efficient planning by showing that an implementation of prioritized sweeping based on small backups achieves a substantial performance improvement over classical implementations.