# Backplay: "Man muss immer umkehren"

arXiv: https://arxiv.org/abs/1807.06919 · HTML: https://mlsanity.com/paper/1807.06919/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2018.

## Built on by (most cited first, 21 of 21)

- [Jump-Start Reinforcement Learning](https://mlsanity.com/paper/2204.02372/) (2022)
- [Generalization to New Sequential Decision Making Tasks with In-Context Learning](https://mlsanity.com/paper/2312.03801/) (2023)
- [Proximal Curriculum for Reinforcement Learning Agents](https://mlsanity.com/paper/2304.12877/) (2023)
- [RL for Reasoning by Adaptively Revealing Rationales](https://mlsanity.com/paper/2506.18110/) (2025)
- [Reverse Forward Curriculum Learning for Extreme Sample and Demonstration Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2405.03379/) (2024)
- [DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robots](https://mlsanity.com/paper/2409.06613/) (2024)
- [Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learning](https://mlsanity.com/paper/2603.15789/) (2026)
- [Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration](https://mlsanity.com/paper/2211.04786/) (2022)
- [Automatic Curricula via Expert Demonstrations](https://mlsanity.com/paper/2106.09159/) (2021)
- [Divide & Conquer Imitation Learning](https://mlsanity.com/paper/2204.07404/) (2022)
- [Learning from Demonstrations via Capability-Aware Goal Sampling](https://mlsanity.com/paper/2601.08731/) (2026)
- [Data-Augmented Game Starts for Accelerating Self-Play Exploration in Imperfect Information Games](https://mlsanity.com/paper/2605.14379/) (2026)
- [CounterPlay: Counterfactual Post-Training for Self-Play Driving Policies](https://mlsanity.com/paper/2609.21617/) (2026)
- [Know your Enemy: Investigating Monte-Carlo Tree Search with Opponent Models in Pommerman](https://mlsanity.com/paper/2305.13206/) (2023)
- [Single-Reset Divide & Conquer Imitation Learning](https://mlsanity.com/paper/2402.09355/) (2024)
- [Playing to Par: Reinforcement Learning for Provably Optimal Quadrilateral Block Decompositions](https://mlsanity.com/paper/2609.32146/) (2026)
- [Bidirectional Voronoi-biased Exploration Curriculum for Reinforcement Learning](https://mlsanity.com/paper/2610.03395/) (2026)
- [Mulligan: Performance-Guided Data Collection for Efficient On-Robot Learning](https://mlsanity.com/paper/2610.05882/) (2026)
- [Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration](https://mlsanity.com/paper/2610.12470/) (2026)
- [Anchor-and-Harvest Curriculum: A Fully Unmasked Non-LLM Curriculum for Craftax](https://mlsanity.com/paper/2610.15328/) (2026)
- [DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation](https://mlsanity.com/paper/2610.19089/) (2026)

## Abstract (arXiv)

Model-free reinforcement learning (RL) requires a large number of trials to learn a good policy, especially in environments with sparse rewards. We explore a method to improve the sample efficiency when we have access to demonstrations. Our approach, Backplay, uses a single demonstration to construct a curriculum for a given task. Rather than starting each training episode in the environment's fixed initial state, we start the agent near the end of the demonstration and move the starting point backwards during the course of training until we reach the initial state. Our contributions are that we analytically characterize the types of environments where Backplay can improve training speed, demonstrate the effectiveness of Backplay both in large grid worlds and a complex four player zero-sum game (Pommerman), and show that Backplay compares favorably to other competitive methods known to improve sample efficiency. This includes reward shaping, behavioral cloning, and reverse curriculum generation.