# You Only Live Once: Single-Life Reinforcement Learning

arXiv: https://arxiv.org/abs/2210.08863 · HTML: https://mlsanity.com/paper/2210.08863/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 75 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 30 of 52)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [Critic Regularized Regression](https://mlsanity.com/paper/2006.15134/) (2020)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [Reinforced Continual Learning](https://mlsanity.com/paper/1805.12369/) (2018)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [Online Meta-Learning](https://mlsanity.com/paper/1902.08438/) (2019)

## Built on by (most cited first, 24 of 24)

- [Continual Learning as Computationally Constrained Reinforcement Learning](https://mlsanity.com/paper/2307.04345/) (2023)
- [Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets](https://mlsanity.com/paper/2402.07320/) (2024)
- [Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2407.09645/) (2024)
- [Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models](https://mlsanity.com/paper/2407.02666/) (2024)
- [Maximum diffusion reinforcement learning](https://mlsanity.com/paper/2309.15293/) (2023)
- [Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment](https://mlsanity.com/paper/2311.01059/) (2023)
- [Details Make a Difference: Object State-Sensitive Neurorobotic Task Planning](https://mlsanity.com/paper/2406.09988/) (2024)
- [Contrast Sets for Evaluating Language-Guided Robot Policies](https://mlsanity.com/paper/2406.13636/) (2024)
- [How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimization](https://mlsanity.com/paper/2309.12671/) (2023)
- [Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems](https://mlsanity.com/paper/2404.15199/) (2024)
- [Prospective Learning: Learning for a Dynamic Future](https://mlsanity.com/paper/2411.00109/) (2024)
- [Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculum](https://mlsanity.com/paper/2305.09943/) (2023)
- [Can language agents be alternatives to PPO? A Preliminary Empirical Study On OpenAI Gym](https://mlsanity.com/paper/2312.03290/) (2023)
- [Prospective Learning in Retrospect](https://mlsanity.com/paper/2507.07965/) (2025)
- [Exploiting Policy Idling for Dexterous Manipulation](https://mlsanity.com/paper/2508.15669/) (2025)
- [Real-Time Reinforcement Learning for Dynamic Tasks with a Parallel Soft Robot](https://mlsanity.com/paper/2509.19525/) (2025)
- [Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces](https://mlsanity.com/paper/2306.01896/) (2023)
- [Enhancing Robotic Manipulation: Harnessing the Power of Multi-Task Reinforcement Learning and Single Life Reinforcement Learning in Meta-World](https://mlsanity.com/paper/2311.12854/) (2023)
- [Life, uh, Finds a Way: Hyperadaptability by Behavioral Search](https://mlsanity.com/paper/2410.01349/) (2024)
- [Requirements for Recognition and Rapid Response to Unfamiliar Events Outside of Agent Design Scope](https://mlsanity.com/paper/2504.12497/) (2025)
- [Optimal control of the future via prospective learning with control](https://mlsanity.com/paper/2511.08717/) (2025)
- [Rule-based High-Level Coaching for Goal-Conditioned Reinforcement Learning in Search-and-Rescue UAV Missions Under Limited-Simulation Training](https://mlsanity.com/paper/2604.26833/) (2026)
- [Intrinsic Vicarious Conditioning for Deep Reinforcement Learning](https://mlsanity.com/paper/2605.12224/) (2026)
- [Nociception as a Control Primitive: Afferent Channels and Nociceptive Memory for Agents Deployed in One Body](https://mlsanity.com/paper/2609.34840/) (2026)

## Abstract (arXiv)

Reinforcement learning algorithms are typically designed to learn a performant policy that can repeatedly and autonomously complete a task, usually starting from scratch. However, in many real-world situations, the goal might not be to learn a policy that can do the task repeatedly, but simply to perform a new task successfully once in a single trial. For example, imagine a disaster relief robot tasked with retrieving an item from a fallen building, where it cannot get direct supervision from humans. It must retrieve this object within one test-time trial, and must do so while tackling unknown obstacles, though it may leverage knowledge it has of the building before the disaster. We formalize this problem setting, which we call single-life reinforcement learning (SLRL), where an agent must complete a task within a single episode without interventions, utilizing its prior experience while contending with some form of novelty. SLRL provides a natural setting to study the challenge of autonomously adapting to unfamiliar situations, and we find that algorithms designed for standard episodic reinforcement learning often struggle to recover from out-of-distribution states in this setting. Motivated by this observation, we propose an algorithm, Q-weighted adversarial learning (QWALE), which employs a distribution matching strategy that leverages the agent's prior experience as guidance in novel situations. Our experiments on several single-life continuous control problems indicate that methods based on our distribution matching formulation are 20-60% more successful because they can more quickly recover from novel states.