mlsanityData to 14 Oct 2026

Dual-Scale World Models for LLM Agents Towards Hard-Exploration Problems

arXiv 2509.24116 ↗First posted 28 Sep 2025cs.CLcited by 2

On the map: World models → World models for language agents

also in 1 more
Abstract (arXiv)

LLM-based agents have seen promising advances, yet they are still limited in "hard-exploration" tasks requiring learning new knowledge through exploration. We present GLoW, a novel approach leveraging dual-scale world models, maintaining a trajectory frontier of high-value discoveries at the global scale, while learning from local trial-and-error in exploration through a Multi-path Advantage Reflection mechanism which infers advantage-based progress signals to guide exploration. To evaluate our framework for hard-exploration, we tackle the Jericho benchmark suite of text-based games, where GLoW achieves a new state-of-theart performance for LLM-based approaches. Compared to state-of-the-art RLbased methods, our approach achieves comparable performance while requiring 100-800x fewer environment interactions.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 27 of its 37 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

27 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment19 Feb
  2. 2021Modeling Worlds in Text17 Jun

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  8. 2023Mastering Diverse Domains through World Modelscited by 1,253
  9. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  10. 2023Cognitive Architectures for Language Agentscited by 429
  11. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  12. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
13 more
  1. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  2. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  3. 2019Interactive Fiction Games: A Colossal Adventurecited by 168
  4. 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 114
  5. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  6. 2020Graph Constrained Reinforcement Learning for Natural Language Action Spacescited by 80
  7. 2024Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Modelscited by 33
  8. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 31
  9. 2020Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learningcited by 27
  10. 2022Multi-Stage Episodic Control for Strategic Exploration in Text Gamescited by 22
  11. 2025Monte Carlo Planning with Large Language Model for Text-Based Game Agentscited by 22
  12. 2025TextQuests: How Good are LLMs at Text-Based Video Games?cited by 12
  13. 2025TALES: Text Adventure Learning Environment Suitecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Joint Agent Memory and Exploration Learning via Novelty Signalscited by 0
  2. 2026Success Leaves Detours: Learning Executable Walkthroughs for Long-Horizon Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Agent LLMs Fail to Explore Each Other
  2. 2026From Task Solving to Robust Real-World Adaptation in LLM Agents
  3. 2026Clearing the Fog: Towards Installing and Refining Proactive Exploration Capabilities in LLM Agents
  4. 2025WorldLLM: Improving LLMs' world modeling using curiosity-driven theory-making
  5. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  6. 2026Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity
  7. 2026Look Before You Leap: Autonomous Exploration for LLM Agents
  8. 2026PhantomEnvironments: Training LLM Agents in Fictional Worlds