mlsanityData to 7 Oct 2026

Autonomous Reinforcement Learning via Subgoal Curricula

arXiv 2107.12931 ↗First posted 27 Jul 2021cs.LG cs.AI cs.ROcited by 32

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) promises to enable autonomous acquisition of complex behaviors for diverse agents. However, the success of current reinforcement learning algorithms is predicated on an often under-emphasised requirement -- each trial needs to start from a fixed initial state distribution. Unfortunately, resetting the environment to its initial state after each trial requires substantial amount of human supervision and extensive instrumentation of the environment which defeats the goal of autonomous acquisition of complex behaviors. In this work, we propose Value-accelerated Persistent Reinforcement Learning (VaPRL), which generates a curriculum of initial states such that the agent can bootstrap on the success of easier tasks to efficiently learn harder tasks. The agent also learns to reach the initial states proposed by the curriculum, minimizing the reliance on human interventions into the learning. We observe that VaPRL reduces the interventions required by three orders of magnitude compared to episodic RL while outperforming prior state-of-the art methods for reset-free RL both in terms of sample efficiency and asymptotic performance on a variety of simulated robotics problems.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 36 of its 63 references that are in the map. First posted on arXiv 27 Jul 2021.

Builds on

36 of 63 references.

References · 36

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  8. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  9. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  10. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  11. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  12. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
24 more
  1. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  2. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  3. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  4. 2017Teacher-Student Curriculum Learningcited by 197
  5. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  6. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  7. 2018Learning to Walk via Deep Reinforcement Learningcited by 153
  8. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  9. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  10. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  11. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  12. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  13. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 103
  14. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  15. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  16. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  17. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  18. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  19. 2018Learning Curriculum Policies for Reinforcement Learningcited by 34
  20. 2017Deep Predictive Policy Training using Reinforcement Learningcited by 30
  21. 2020Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learningcited by 30
  22. 2020Continual Learning of Control Primitives: Skill Discovery via Reset-Gamescited by 29
  23. 2017Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learningcited by 27
  24. 2017TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlowcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 11; 2024: 4; 2025: 1; 2026: 7.

32 papers.

Sort:

By area:

  1. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  2. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
  3. 2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 32
  4. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  5. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  6. 2022Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learningcited by 28
  7. 2022You Only Live Once: Single-Life Reinforcement Learningcited by 23
  8. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  9. 2022When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learningcited by 20
  10. 2021Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulationcited by 13
  11. 2023REBOOT: Reuse Data for Bootstrapping Efficient Real-World Dexterous Manipulationcited by 12
  12. 2023Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedbackcited by 12
20 more
  1. 2022Automating Reinforcement Learning with Example-based Resetscited by 11
  2. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agentscited by 10
  3. 2022Hierarchical Policy Blending As Optimal Transportcited by 4
  4. 2024Intelligent Switching for Reset-Free RLcited by 4
  5. 2026UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learningcited by 4
  6. 2023When Learning Is Out of Reach, Reset: Generalization in Autonomous Visuomotor Reinforcement Learningcited by 3
  7. 2024NeoRL: Efficient Exploration for Nonepisodic RLcited by 3
  8. 2024Reset-free Reinforcement Learning with World Modelscited by 3
  9. 2023CRISP: Curriculum Inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learningcited by 2
  10. 2023Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculumcited by 2
  11. 2023L-SA: Learning Under-Explored Targets in Multi-Target Reinforcement Learningcited by 2
  12. 2023SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situationscited by 2
  13. 2026What Matters for Simulation to Online Reinforcement Learning on Real Robotscited by 2
  14. 2026Tether: Autonomous Functional Play with Correspondence-Driven Trajectory Warpingcited by 2
  15. 2023Self-Supervised Curriculum Generation for Autonomous Reinforcement Learning without Task-Specific Knowledgecited by 1
  16. 2024Autonomous Algorithm for Training Autonomous Vehicles with Minimal Human Interventioncited by 1
  17. 2025Training RL Agents for Multi-Objective Network Defense Taskscited by 1
  18. 2026Reset-Free Reinforcement Learning for Real-World Agile Driving: An Empirical Studycited by 0
  19. 2026REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliffcited by 0
  20. 2026Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Modelscited by 0