# Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective

arXiv: https://arxiv.org/abs/2510.15047 · HTML: https://mlsanity.com/paper/2510.15047/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2504.20073/) (2025)
- [Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs](https://mlsanity.com/paper/2506.14245/) (2025)
- [A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models](https://mlsanity.com/paper/2503.23350/) (2025)
- [Large Language Models as Generalizable Policies for Embodied Tasks](https://mlsanity.com/paper/2310.17722/) (2023)
- [A Survey of Deep Reinforcement Learning in Video Games](https://mlsanity.com/paper/1912.10944/) (2019)
- [Large Language Models for Planning: A Comprehensive and Systematic Survey](https://mlsanity.com/paper/2505.19683/) (2025)

## Built on by (most cited first, 28 of 28)

- [Qwen-AgentWorld: Language World Models for General Agents](https://mlsanity.com/paper/2606.24597/) (2026)
- [WebWorld: A Large-Scale World Model for Web Agent Training](https://mlsanity.com/paper/2602.14721/) (2026)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [RAGEN-2: Reasoning Collapse in Agentic RL](https://mlsanity.com/paper/2604.06268/) (2026)
- [Self-Improvements in Modern Agentic Systems: A Survey](https://mlsanity.com/paper/2607.13104/) (2026)
- [Learning How to Remember: A Meta-Cognitive Management Method for Structured and Transferable Agent Memory](https://mlsanity.com/paper/2601.07470/) (2026)
- [Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Models](https://mlsanity.com/paper/2601.08955/) (2026)
- [Beyond Next-Observation Prediction: Agent-Authored World Modeling for Sequential Decision Making](https://mlsanity.com/paper/2606.25421/) (2026)
- [OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation](https://mlsanity.com/paper/2604.10866/) (2026)
- [Aligning Agentic World Models via Knowledgeable Experience Learning](https://mlsanity.com/paper/2601.13247/) (2026)
- [MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games](https://mlsanity.com/paper/2603.09022/) (2026)
- [Distilling Game Code World Model Generation into Lightweight Large Language Models](https://mlsanity.com/paper/2605.24375/) (2026)
- [Bridging the Agent-World Gap: Text World Models for LLM-based Agents](https://mlsanity.com/paper/2606.09032/) (2026)
- [Scaling GUI Agents with Visual State Transitions](https://mlsanity.com/paper/2607.24112/) (2026)
- [Active Intelligence in Video Avatars via Closed-loop World Modeling](https://mlsanity.com/paper/2512.20615/) (2025)
- [Internalizing the Future: A Unified Agentic Training Paradigm for World Model Planning](https://mlsanity.com/paper/2606.27483/) (2026)
- [The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation](https://mlsanity.com/paper/2607.24720/) (2026)
- [State2State: Environment-Derived Mid-Training for LLM Agents](https://mlsanity.com/paper/2608.04934/) (2026)
- [EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning](https://mlsanity.com/paper/2608.06197/) (2026)
- [Language-based Trial and Error Falls Behind in the Era of Experience](https://mlsanity.com/paper/2601.21754/) (2026)
- [Internalizing Agency from Reflective Experience](https://mlsanity.com/paper/2603.16843/) (2026)
- [What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity](https://mlsanity.com/paper/2605.03782/) (2026)
- [Cross-Environment Neural Reranking for Sample-Efficient Action Selection in Text-Based Agents](https://mlsanity.com/paper/2606.02204/) (2026)
- [How do World Models and Policies Compose in LLM Agents? A Joint Spectral and Behavioral Account](https://mlsanity.com/paper/2608.30067/) (2026)
- [WMLLM: Self-Evolving Optimization Agents via Predict-Then-Act World Modeling](https://mlsanity.com/paper/2609.01608/) (2026)
- [Does Learning to Predict the World Help Agents Act? Auditing World-Model Post-Training](https://mlsanity.com/paper/2609.33335/) (2026)
- [Schema: Discovering Unknown Environments via Agentic Program Induction](https://mlsanity.com/paper/2609.39140/) (2026)
- [WOVEN: Weaving Visual World Modeling into Multimodal LLMs](https://mlsanity.com/paper/2610.12417/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) as agents often fail to improve in new environments. We identify and characterize a failure mode we call exploration collapse: under reinforcement learning (RL) in environments whose states are unfamiliar to the policy, Pass@k, the probability that at least one of k sampled trajectories succeeds, drops markedly over training even as Pass@1 edges up, revealing increasingly brittle exploration; environments closer to the pretraining distribution show no such decline. We trace this collapse to weak grounding in environment states and dynamics, and study a simple remedy: explicitly teaching the agent to estimate the current state and predict its transitions before optimizing for reward. We instantiate it as SPA, an explore-then-exploit recipe that cold-starts the policy with a Self-Experience supervised finetuning (SFT) stage, collecting the model's own interaction trajectories and supervising state and next-state prediction, and then runs standard RL. The resulting world model serves as a grounded initialization for RL rather than an inference-time planner. Across unseen environments, SPA consistently and substantially improves over vanilla RL: for example, it raises the Sokoban success rate from 25.6% to 59.8% on Qwen2.5-1.5B-Instruct, letting sub-3B models surpass a 20B baseline on these tasks. Controlled studies indicate that the gains track four factors: grounded state representations, explicit transition modeling, self-experience trajectories from a sufficiently strong exploration policy, and adequate coverage of transition data.