mlsanityData to 7 Oct 2026

You Only Live Once: Single-Life Reinforcement Learning

arXiv 2210.08863 ↗First posted 17 Oct 2022cs.LG cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning algorithms are typically designed to learn a performant policy that can repeatedly and autonomously complete a task, usually starting from scratch. However, in many real-world situations, the goal might not be to learn a policy that can do the task repeatedly, but simply to perform a new task successfully once in a single trial. For example, imagine a disaster relief robot tasked with retrieving an item from a fallen building, where it cannot get direct supervision from humans. It must retrieve this object within one test-time trial, and must do so while tackling unknown obstacles, though it may leverage knowledge it has of the building before the disaster. We formalize this problem setting, which we call single-life reinforcement learning (SLRL), where an agent must complete a task within a single episode without interventions, utilizing its prior experience while contending with some form of novelty. SLRL provides a natural setting to study the challenge of autonomously adapting to unfamiliar situations, and we find that algorithms designed for standard episodic reinforcement learning often struggle to recover from out-of-distribution states in this setting. Motivated by this observation, we propose an algorithm, Q-weighted adversarial learning (QWALE), which employs a distribution matching strategy that leverages the agent's prior experience as guidance in novel situations. Our experiments on several single-life continuous control problems indicate that methods based on our distribution matching formulation are 20-60% more successful because they can more quickly recover from novel states.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 52 of its 75 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

52 of 75 references.

References · 52

  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  8. 2016Progressive Neural Networkscited by 1,332
  9. 2018Experience Replay for Continual Learningcited by 1,074
  10. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
38 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  2. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  5. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  6. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  7. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  8. 2018Maximum a Posteriori Policy Optimisationcited by 318
  9. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  10. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  11. 2020Critic Regularized Regressioncited by 269
  12. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  13. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  14. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  15. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  16. 2018Reinforced Continual Learningcited by 186
  17. 2019Active Domain Randomizationcited by 161
  18. 2019Online Meta-Learningcited by 137
  19. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  20. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  21. 2019Imitation Learning from Imperfect Demonstrationcited by 112
  22. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  23. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  24. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  25. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  26. 2019Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processescited by 81
  27. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  28. 2021Off-Policy Imitation Learning from Observationscited by 70
  29. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  30. 2020Deep Reinforcement Learning amidst Lifelong Non-Stationaritycited by 42
  31. 2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
  32. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  33. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  34. 2019Continual Reinforcement Learning in 3D Non-stationary Environmentscited by 21
  35. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  36. 2019Adversarial Imitation Learning from Incomplete Demonstrationscited by 16
  37. 2022Demonstration-Bootstrapped Autonomous Practicing via Multi-Task Reinforcement Learningcited by 15
  38. 2022Automating Reinforcement Learning with Example-based Resetscited by 11

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 8; 2025: 4; 2026: 3.

23 papers.

Sort:

By area:

  1. 2023Continual Learning as Computationally Constrained Reinforcement Learningcited by 35
  2. 2024Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Setscited by 25
  3. 2024Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Modelscited by 15
  4. 2024Hamilton-Jacobi Reachability in Reinforcement Learning: A Surveycited by 11
  5. 2023Maximum diffusion reinforcement learningcited by 8
  6. 2023Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deploymentcited by 7
  7. 2024Details Make a Difference: Object State-Sensitive Neurorobotic Task Planningcited by 7
  8. 2024Contrast Sets for Evaluating Language-Guided Robot Policiescited by 7
  9. 2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
  10. 2024Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systemscited by 6
  11. 2024Prospective Learning: Learning for a Dynamic Futurecited by 4
  12. 2023Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculumcited by 2
11 more
  1. 2023Can language agents be alternatives to PPO? A Preliminary Empirical Study On OpenAI Gymcited by 2
  2. 2025Prospective Learning in Retrospectcited by 1
  3. 2025Exploiting Policy Idling for Dexterous Manipulationcited by 1
  4. 2023Learning to Stabilize Online Reinforcement Learning in Unbounded State Spacescited by 0
  5. 2023Enhancing Robotic Manipulation: Harnessing the Power of Multi-Task Reinforcement Learning and Single Life Reinforcement Learning in Meta-Worldcited by 0
  6. 2024Life, uh, Finds a Way: Hyperadaptability by Behavioral Searchcited by 0
  7. 2025Requirements for Recognition and Rapid Response to Unfamiliar Events Outside of Agent Design Scopecited by 0
  8. 2025Optimal control of the future via prospective learning with controlcited by 0
  9. 2026Rule-based High-Level Coaching for Goal-Conditioned Reinforcement Learning in Search-and-Rescue UAV Missions Under Limited-Simulation Trainingcited by 0
  10. 2026Intrinsic Vicarious Conditioning for Deep Reinforcement Learningcited by 0
  11. 2026Nociception as a Control Primitive: Afferent Channels and Nociceptive Memory for Agents Deployed in One Bodycited by 0