# Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning

arXiv: https://arxiv.org/abs/1711.06782 · HTML: https://mlsanity.com/paper/1711.06782/

## Key facts

- Cited by 90 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2017.

## Built on by (most cited first, 30 of 90)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning](https://mlsanity.com/paper/2401.16013/) (2024)
- [Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention](https://mlsanity.com/paper/2104.11203/) (2021)
- [Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.03400/) (2021)
- [Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation](https://mlsanity.com/paper/2202.06558/) (2022)
- [Safe Reinforcement Learning by Imagining the Near Future](https://mlsanity.com/paper/2202.07789/) (2022)
- [RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning](https://mlsanity.com/paper/2510.14830/) (2025)
- [Constrained Policy Optimization via Bayesian World Models](https://mlsanity.com/paper/2201.09802/) (2022)
- [Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning](https://mlsanity.com/paper/2310.15145/) (2023)
- [Safe Reinforcement Learning Using Advantage-Based Intervention](https://mlsanity.com/paper/2106.09110/) (2021)
- [Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations](https://mlsanity.com/paper/2108.01846/) (2021)
- [Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation](https://mlsanity.com/paper/2107.13545/) (2021)
- [Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning](https://mlsanity.com/paper/2303.01488/) (2023)
- [Autonomous Reinforcement Learning via Subgoal Curricula](https://mlsanity.com/paper/2107.12931/) (2021)
- [Autonomous Reinforcement Learning: Formalism and Benchmarking](https://mlsanity.com/paper/2112.09605/) (2021)
- [Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning](https://mlsanity.com/paper/2207.04703/) (2022)
- [Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms](https://mlsanity.com/paper/2310.03225/) (2023)
- [Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees](https://mlsanity.com/paper/2201.08355/) (2022)
- [You Only Live Once: Single-Life Reinforcement Learning](https://mlsanity.com/paper/2210.08863/) (2022)
- [Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance](https://mlsanity.com/paper/2212.09902/) (2022)
- [Asynchronous Reinforcement Learning for Real-Time Control of Physical Robots](https://mlsanity.com/paper/2203.12759/) (2022)
- [A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning](https://mlsanity.com/paper/2205.05212/) (2022)
- [When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learning](https://mlsanity.com/paper/2210.10765/) (2022)
- [Dealing with the Unknown: Pessimistic Offline Reinforcement Learning](https://mlsanity.com/paper/2111.05440/) (2021)
- [C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks](https://mlsanity.com/paper/2110.12080/) (2021)
- [Transferring Multi-Agent Reinforcement Learning Policies for Autonomous Driving using Sim-to-Real](https://mlsanity.com/paper/2203.11653/) (2022)
- [FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing](https://mlsanity.com/paper/2304.09831/) (2023)
- [REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulation](https://mlsanity.com/paper/2309.03322/) (2023)

## Abstract (arXiv)

Deep reinforcement learning algorithms can learn complex behavioral skills, but real-world application of these methods requires a large amount of experience to be collected by the agent. In practical settings, such as robotics, this involves repeatedly attempting a task, resetting the environment between each attempt. However, not all tasks are easily or automatically reversible. In practice, this learning process requires extensive human intervention. In this work, we propose an autonomous method for safe and efficient reinforcement learning that simultaneously learns a forward and reset policy, with the reset policy resetting the environment for a subsequent attempt. By learning a value function for the reset policy, we can automatically determine when the forward policy is about to enter a non-reversible state, providing for uncertainty-aware safety aborts. Our experiments illustrate that proper use of the reset policy can greatly reduce the number of manual resets required to learn a task, can reduce the number of unsafe actions that lead to non-reversible states, and can automatically induce a curriculum.