mlsanityData to 13 Oct 2026

Planning to Explore via Self-Supervised World Models

arXiv 2005.05960 ↗First posted 12 May 2020cs.LG cs.AI cs.CVcited by 367

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge. We present Plan2Explore, a self-supervised reinforcement learning agent that tackles both these challenges through a new approach to self-supervised exploration and fast adaptation to new tasks, which need not be known during exploration. During exploration, unlike prior methods which retrospectively compute the novelty of observations after the agent has already reached them, our agent acts efficiently by leveraging planning to seek out expected future novelty. After exploration, the agent quickly adapts to multiple downstream tasks in a zero or a few-shot manner. We evaluate on challenging control tasks from high-dimensional image inputs. Without any training supervision or task-specific interaction, Plan2Explore outperforms prior self-supervised exploration methods, and in fact, almost matches the performances oracle which has access to rewards. Videos and code at https://ramanans1.github.io/plan2explore/

Key facts. Cited by 367 papers in the map (data through 13 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 4 of its 62 references that are in the map. First posted on arXiv 12 May 2020.

Builds on

4 of 62 references.

Closest ancestors · 3

Same direction, newest first

  1. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May
  3. 2018World Models27 Mar

Other references · 1

  1. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,241

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2020: 4; 2021: 52; 2022: 62; 2023: 80; 2024: 52; 2025: 65; 2026: 52.

367 papers.

Sort:

By area:

  1. 2020Mastering Atari with Discrete World Modelscited by 996
  2. 2023Reasoning with Language Model is Planning with World Modelcited by 851
  3. 2022Temporal Difference Learning for Model Predictive Controlcited by 384
  4. 2022DayDreamer: World Models for Physical Robot Learningcited by 375
  5. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 369
  6. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 311
  7. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 240
  8. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 194
  9. 2021Reinforcement Learning with Prototypical Representationscited by 179
  10. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  11. 2023Structured World Models from Human Videoscited by 145
  12. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 145
38 more
  1. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  2. 2021Discovering and Achieving Goals via World Modelscited by 131
  3. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  4. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  5. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  6. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  7. 2021Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Designcited by 105
  8. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  9. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 103
  10. 2022Deep Hierarchical Planning from Pixelscited by 99
  11. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 94
  12. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  13. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 83
  14. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  15. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 82
  16. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 79
  17. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  18. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 76
  19. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
  20. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
  21. 2022Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigationcited by 66
  22. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  23. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  24. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  25. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 59
  26. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  27. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  28. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  29. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  30. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  31. 2022Skill-based Model-based Reinforcement Learningcited by 50
  32. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  33. 2021Learning to Map for Active Semantic Goal Navigationcited by 48
  34. 2025Self-Questioning Language Modelscited by 48
  35. 2026World Action Models: The Next Frontier in Embodied AIcited by 48
  36. 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 47
  37. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  38. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45

50 most cited of 367 All 367 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptation
  2. 2025Curiosity-Driven Imagination: Discovering Plan Operators and Learning Associated Policies for Open-World Adaptation
  3. 2019Learning Exploration Policies for Navigation
  4. 2026SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents
  5. 2026Look Before You Leap: Autonomous Exploration for LLM Agents
  6. 2026ProPlay: Procedural World Models for Self-Evolving LLM Agents
  7. 2020Latent Skill Planning for Exploration and Transfer
  8. 2019Supervise Thyself: Examining Self-Supervised Representations in Interactive Environments