# World Modelling Improves Language Model Agents

arXiv: https://arxiv.org/abs/2506.02918 · HTML: https://mlsanity.com/paper/2506.02918/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint](https://mlsanity.com/paper/2312.11456/) (2023)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [Direct Language Model Alignment from Online AI Feedback](https://mlsanity.com/paper/2402.04792/) (2024)
- [ToolACE: Winning the Points of LLM Function Calling](https://mlsanity.com/paper/2409.00920/) (2024)
- [APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets](https://mlsanity.com/paper/2406.18518/) (2024)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning](https://mlsanity.com/paper/2410.02089/) (2024)
- [Understanding the performance gap between online and offline alignment algorithms](https://mlsanity.com/paper/2405.08448/) (2024)
- [Reinforcement Learning for Long-Horizon Interactive LLM Agents](https://mlsanity.com/paper/2502.01600/) (2025)
- [Self-rewarding correction for mathematical reasoning](https://mlsanity.com/paper/2502.19613/) (2025)
- [Direct Multi-Turn Preference Optimization for Language Agents](https://mlsanity.com/paper/2406.14868/) (2024)
- [Command A: An Enterprise-Ready Large Language Model](https://mlsanity.com/paper/2504.00698/) (2025)

## Built on by (most cited first, 14 of 14)

- [Agent Learning via Early Experience](https://mlsanity.com/paper/2510.08558/) (2025)
- [EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis](https://mlsanity.com/paper/2601.05808/) (2026)
- [Qwen-AgentWorld: Language World Models for General Agents](https://mlsanity.com/paper/2606.24597/) (2026)
- [Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agents](https://mlsanity.com/paper/2601.22311/) (2026)
- [Policy and World Modeling Co-Training for Language Agents](https://mlsanity.com/paper/2606.02388/) (2026)
- [Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond](https://mlsanity.com/paper/2604.22748/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Beyond Next-Observation Prediction: Agent-Authored World Modeling for Sequential Decision Making](https://mlsanity.com/paper/2606.25421/) (2026)
- [ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation](https://mlsanity.com/paper/2602.01709/) (2026)
- [EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning](https://mlsanity.com/paper/2608.06197/) (2026)
- [Quo Vadis, World Modeling?](https://mlsanity.com/paper/2608.02713/) (2026)
- [Agent-Editing World Model: Rethinking World Modeling for LLM Agents](https://mlsanity.com/paper/2609.28416/) (2026)
- [Choosing Before Acting: Comparative Value Estimation for Long-Horizon Tool-Use Agents](https://mlsanity.com/paper/2610.02330/) (2026)
- [Software World Models: From Consequence Prediction to Decision Value](https://mlsanity.com/paper/2610.04940/) (2026)

## Abstract (arXiv)

Tool use in stateful environments presents unique challenges for large language models (LLMs), where existing test-time compute strategies relying on repeated trials in the environment are impractical. We propose dynamics modelling (DyMo), a method that augments LLMs with a state prediction capability alongside function calling during post-training. This enables LLMs to predict the future states of their actions through an internal environment model. On the Berkeley Function Calling Leaderboard V2, DyMo improves success rates and significantly reduces hallucinations. We further integrate the internal environment model into self-verification sampling (SVS), and show that this substantially improves pass^k over number of trials k, and allows the model to refuse unreliable outputs. Together, DyMo and SVS greatly enhance the effectiveness and reliability of LLMs for tool use. We believe this work charts a path towards scalable planning RL methods for LLM inference without repeatedly querying the oracle environment.