# Automating Reinforcement Learning with Example-based Resets

arXiv: https://arxiv.org/abs/2204.02041 · HTML: https://mlsanity.com/paper/2204.02041/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 30 references that are in the map.
- First posted on arXiv 5 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [The Ingredients of Real-World Robotic Reinforcement Learning](https://mlsanity.com/paper/2004.12570/) (2020)
- [Learning to Walk in the Real World with Minimal Human Effort](https://mlsanity.com/paper/2002.08550/) (2020)
- [AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos](https://mlsanity.com/paper/1912.04443/) (2019)
- [Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention](https://mlsanity.com/paper/2104.11203/) (2021)
- [Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning](https://mlsanity.com/paper/1711.06782/) (2017)
- [Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition](https://mlsanity.com/paper/1805.11686/) (2018)
- [Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement](https://mlsanity.com/paper/2002.11089/) (2020)
- [Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly](https://mlsanity.com/paper/1910.13675/) (2019)
- [Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification](https://mlsanity.com/paper/2103.12656/) (2021)
- [Autonomous Reinforcement Learning via Subgoal Curricula](https://mlsanity.com/paper/2107.12931/) (2021)
- [Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2004.12974/) (2020)
- [Continual Learning of Control Primitives: Skill Discovery via Reset-Games](https://mlsanity.com/paper/2011.05286/) (2020)
- [Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search](https://mlsanity.com/paper/1610.00673/) (2016)
- [There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning](https://mlsanity.com/paper/2106.04480/) (2021)

## Built on by (most cited first, 12 of 12)

- [Behavior Generation with Latent Actions](https://mlsanity.com/paper/2403.03181/) (2024)
- [Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning](https://mlsanity.com/paper/2303.01488/) (2023)
- [You Only Live Once: Single-Life Reinforcement Learning](https://mlsanity.com/paper/2210.08863/) (2022)
- [IMLE Policy: Fast and Sample Efficient Visuomotor Policy Learning via Implicit Maximum Likelihood Estimation](https://mlsanity.com/paper/2502.12371/) (2025)
- [UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learning](https://mlsanity.com/paper/2606.12372/) (2026)
- [Intelligent Switching for Reset-Free RL](https://mlsanity.com/paper/2405.01684/) (2024)
- [Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum](https://mlsanity.com/paper/2305.09943/) (2023)
- [Safety-Driven Deep Reinforcement Learning Framework for Cobots: A Sim2Real Approach](https://mlsanity.com/paper/2407.02231/) (2024)
- [Reset-free Reinforcement Learning with World Models](https://mlsanity.com/paper/2408.09807/) (2024)
- [Self-Supervised Curriculum Generation for Autonomous Reinforcement Learning without Task-Specific Knowledge](https://mlsanity.com/paper/2311.09195/) (2023)
- [Real-Time Reinforcement Learning for Dynamic Tasks with a Parallel Soft Robot](https://mlsanity.com/paper/2509.19525/) (2025)
- [Learning Without Time-Based Embodiment Resets in Soft-Actor Critic](https://mlsanity.com/paper/2512.06252/) (2025)

## Abstract (arXiv)

Deep reinforcement learning has enabled robots to learn motor skills from environmental interactions with minimal to no prior knowledge. However, existing reinforcement learning algorithms assume an episodic setting, in which the agent resets to a fixed initial state distribution at the end of each episode, to successfully train the agents from repeated trials. Such reset mechanism, while trivial for simulated tasks, can be challenging to provide for real-world robotics tasks. Resets in robotic systems often require extensive human supervision and task-specific workarounds, which contradicts the goal of autonomous robot learning. In this paper, we propose an extension to conventional reinforcement learning towards greater autonomy by introducing an additional agent that learns to reset in a self-supervised manner. The reset agent preemptively triggers a reset to prevent manual resets and implicitly imposes a curriculum for the forward agent. We apply our method to learn from scratch on a suite of simulated and real-world continuous control tasks and demonstrate that the reset agent successfully learns to reduce manual resets whilst also allowing the forward policy to improve gradually over time.