mlsanityData to 7 Oct 2026

World Models with Hints of Large Language Models for Goal Achieving

arXiv 2406.07381 ↗First posted 11 Jun 2024cs.AI cs.LGcited by 6

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Reinforcement learning struggles in the face of long-horizon tasks and sparse goals due to the difficulty in manual reward specification. While existing methods address this by adding intrinsic rewards, they may fail to provide meaningful guidance in long-horizon decision-making tasks with large state and action spaces, lacking purposeful exploration. Inspired by human cognition, we propose a new multi-modal model-based RL approach named Dreaming with Large Language Models (DLLM). DLLM integrates the proposed hinting subgoals from the LLMs into the model rollouts to encourage goal discovery and reaching in challenging tasks. By assigning higher intrinsic rewards to samples that align with the hints outlined by the language model during model rollouts, DLLM guides the agent toward meaningful and efficient exploration. Extensive experiments demonstrate that the DLLM outperforms recent methods in various challenging, sparse-reward environments such as HomeGrid, Crafter, and Minecraft by 27.7%, 21.1%, and 9.9%, respectively.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 53 of its 77 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

53 of 77 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Mastering Diverse Domains through World Models10 Jan
  2. 2020Mastering Atari with Discrete World Models5 Oct
  3. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  4. 2018World Models27 Mar

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  10. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
37 more
  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  3. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  4. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  5. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  6. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  7. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  8. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  9. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  10. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  11. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  12. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  13. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  14. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  15. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  16. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  17. 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
  18. 2020Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Surveycited by 83
  19. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  20. 2023Learning to Model the World with Languagecited by 66
  21. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  22. 2013Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robotscited by 56
  23. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  24. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  25. 2018GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithmscited by 40
  26. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  27. 2018Investigating Human Priors for Playing Video Gamescited by 39
  28. 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
  29. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  30. 2023Evolutionary Reinforcement Learning: A Surveycited by 24
  31. 2017Efficient exploration with Double Uncertain Value Networkscited by 15
  32. 2023COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RLcited by 14
  33. 2019Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Explorationcited by 11
  34. 2023AdaRefiner: Refining Decisions of Language Models with Adaptive Feedbackcited by 10
  35. 2023Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learningcited by 6
  36. 2018Learning Temporal Strategic Relationships using Generative Adversarial Imitation Learningcited by 3
  37. 2022Goal Exploration Augmentation via Pre-trained Skills for Sparse-Reward Long-Horizon Goal-Conditioned Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  2. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9
  3. 2026Self-Improving World Modelling with Latent Actionscited by 5
  4. 2026SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Groundingcited by 4
  5. 2025SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modelingcited by 3
  6. 2025Context and Diversity Matter: The Emergence of In-Context Learning in World Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2023Vision-Language Models as a Source of Rewards
  2. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  3. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling
  4. 2024Words as Beacons: Guiding RL Agents with High-Level Language Prompts
  5. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  6. 2025GoalLadder: Incremental Goal Discovery with Vision-Language Models
  7. 2025Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models
  8. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback