mlsanityData to 7 Oct 2026

Reset-free Reinforcement Learning with World Models

arXiv 2408.09807 ↗First posted 19 Aug 2024cs.AIcited by 3

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Reinforcement learning (RL) is an appealing paradigm for training intelligent agents, enabling policy acquisition from the agent's own autonomously acquired experience. However, the training process of RL is far from automatic, requiring extensive human effort to reset the agent and environments. To tackle the challenging reset-free setting, we first demonstrate the superiority of model-based (MB) RL methods in such setting, showing that a straightforward adaptation of MBRL can outperform all the prior state-of-the-art methods while requiring less supervision. We then identify limitations inherent to this direct extension and propose a solution called model-based reset-free (MoReFree) agent, which further enhances the performance. MoReFree adapts two key mechanisms, exploration and policy learning, to handle reset-free tasks by prioritizing task-relevant states. It exhibits superior data-efficiency across various reset-free tasks without access to environmental reward or demonstrations while significantly outperforming privileged baselines that require supervision. Our findings suggest model-based methods hold significant promise for reducing human effort in RL. Website: https://yangzhao-666.github.io/morefree

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 36 of its 36 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

36 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Mastering Atari with Discrete World Models5 Oct
  2. 2020Planning to Explore via Self-Supervised World Models12 May
  3. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec

Other references · 33

  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  2. 2018Exploration by Random Network Distillationcited by 966
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  4. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  5. 2020First return, then explorecited by 256
  6. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  7. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  8. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  9. 2021Discovering and Achieving Goals via World Modelscited by 131
  10. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  11. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  12. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
21 more
  1. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  2. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  3. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  4. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  5. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  6. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  7. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  8. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  9. 2013Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robotscited by 56
  10. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  11. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  12. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  13. 2020Continual Learning of Control Primitives: Skill Discovery via Reset-Gamescited by 29
  14. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  15. 2023Planning Goals for Explorationcited by 28
  16. 2016Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Searchcited by 25
  17. 2018Many-Goals Reinforcement Learningcited by 22
  18. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  19. 2022Automating Reinforcement Learning with Example-based Resetscited by 11
  20. 2019Adaptive Online Planning for Continual Lifelong Learningcited by 7
  21. 2023Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculumcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learningcited by 2
  2. 2025Learning Without Time-Based Embodiment Resets in Soft-Actor Criticcited by 1
  3. 2026REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliffcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020MOReL : Model-Based Offline Reinforcement Learning
  2. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning
  3. 2019Reinforcement learning with world model
  4. 2020MOPO: Model-based Offline Policy Optimization
  5. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis
  6. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model
  7. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning
  8. 2024Dataset Reset Policy Optimization for RLHF