mlsanityData to 7 Oct 2026

Learning World Models for Unconstrained Goal Navigation

arXiv 2411.02446 ↗First posted 3 Nov 2024cs.LG cs.AI cs.ROcited by 6

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Learning world models offers a promising avenue for goal-conditioned reinforcement learning with sparse rewards. By allowing agents to plan actions or exploratory goals without direct interaction with the environment, world models enhance exploration efficiency. The quality of a world model hinges on the richness of data stored in the agent's replay buffer, with expectations of reasonable generalization across the state space surrounding recorded trajectories. However, challenges arise in generalizing learned world models to state transitions backward along recorded trajectories or between states across different trajectories, hindering their ability to accurately model real-world dynamics. To address these challenges, we introduce a novel goal-directed exploration algorithm, MUN (short for "World Models for Unconstrained Goal Navigation"). This algorithm is capable of modeling state transitions between arbitrary subgoal states in the replay buffer, thereby facilitating the learning of policies to navigate between any "key" states. Experimental results demonstrate that MUN strengthens the reliability of world models and significantly improves the policy's capacity to generalize across new goal settings.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 52 of its 60 references that are in the map. First posted on arXiv 3 Nov 2024.

Builds on

52 of 60 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Control25 Oct
  2. 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning14 Oct
  3. 2023Mastering Diverse Domains through World Models10 Jan
  4. 2020Mastering Atari with Discrete World Models5 Oct
  5. 2020Planning to Explore via Self-Supervised World Models12 May
  6. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  7. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun
  8. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees10 Jul
  9. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models30 May
  10. 2018Model-Ensemble Trust-Region Policy Optimization28 Feb
  11. 2017Value Prediction Network11 Jul

Other references · 41

  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  9. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  11. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  12. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
29 more
  1. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  2. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  3. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  4. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  5. 2021Discovering and Achieving Goals via World Modelscited by 131
  6. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  7. 2018Model-Based Active Explorationcited by 105
  8. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  9. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  10. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  11. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  12. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  13. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  14. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  15. 2019Towards Practical Multi-Object Manipulation using Relational Reinforcement Learningcited by 54
  16. 2019Exploration via Hindsight Goal Generationcited by 53
  17. 2020Goal-Aware Prediction: Learning to Model What Matterscited by 45
  18. 2023Planning Goals for Explorationcited by 28
  19. 2019Learning from Trajectories via Subgoal Discoverycited by 27
  20. 2022Multi-Stage Episodic Control for Strategic Exploration in Text Gamescited by 22
  21. 2020PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goalscited by 21
  22. 2021When should agents explore?cited by 20
  23. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  24. 2022The Benefits of Model-Based Generalization in Reinforcement Learningcited by 17
  25. 2023Curious Replay for Model-based Adaptationcited by 16
  26. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  27. 2021On Effective Scheduling of Model-based Reinforcement Learningcited by 13
  28. 2019Curiosity-Driven Multi-Criteria Hindsight Experience Replaycited by 11
  29. 2021Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Taskscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agentscited by 10
  2. 2025Context and Diversity Matter: The Emergence of In-Context Learning in World Modelscited by 3
  3. 2025Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectoriescited by 3
  4. 2025Deep Active Inference with Diffusion Policy and Multiple Timescale World Model for Real-World Exploration and Navigationcited by 1
  5. 2026Learning from Demonstrations via Capability-Aware Goal Samplingcited by 1
  6. 2026Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026World Model for Robot Learning: A Comprehensive Survey
  2. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration
  3. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization
  4. 2022The Effectiveness of World Models for Continual Reinforcement Learning
  5. 2024Boosting Efficiency in Task-Agnostic Exploration through Causal Knowledge
  6. 2026World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning
  7. 2025RLVR-World: Training World Models with Reinforcement Learning
  8. 2024PWM: Policy Learning with Multi-Task World Models