mlsanityData to 14 Oct 2026

World Modelling Improves Language Model Agents

arXiv 2506.02918 ↗First posted 3 Jun 2025cs.AI cs.LGcited by 14

On the map: World models → World models for language agents

also in 2 more
Abstract (arXiv)

Tool use in stateful environments presents unique challenges for large language models (LLMs), where existing test-time compute strategies relying on repeated trials in the environment are impractical. We propose dynamics modelling (DyMo), a method that augments LLMs with a state prediction capability alongside function calling during post-training. This enables LLMs to predict the future states of their actions through an internal environment model. On the Berkeley Function Calling Leaderboard V2, DyMo improves success rates and significantly reduces hallucinations. We further integrate the internal environment model into self-verification sampling (SVS), and show that this substantially improves pass^k over number of trials k, and allows the model to refuse unreliable outputs. Together, DyMo and SVS greatly enhance the effectiveness and reliability of LLMs for tool use. We believe this work charts a path towards scalable planning RL methods for LLM inference without repeatedly querying the oracle environment.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 34 of its 50 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

34 of 50 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  12. 2020Learning to summarize from human feedbackcited by 2,588
22 more
  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  2. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  5. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  6. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  8. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  9. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  10. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  11. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
  12. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  13. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  14. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  15. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 108
  16. 2025Self-rewarding correction for mathematical reasoningcited by 70
  17. 2024Direct Multi-Turn Preference Optimization for Language Agentscited by 64
  18. 2025Command A: An Enterprise-Ready Large Language Modelcited by 59
  19. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 50
  20. 2024Mastering Board Games by External and Internal Planning with Language Modelscited by 31
  21. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 29
  22. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2025Agent Learning via Early Experiencecited by 85
  2. 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesiscited by 63
  3. 2026Qwen-AgentWorld: Language World Models for General Agentscited by 28
  4. 2026Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agentscited by 23
  5. 2026Policy and World Modeling Co-Training for Language Agentscited by 15
  6. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 14
  7. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  8. 2026Beyond Next-Observation Prediction: Agent-Authored World Modeling for Sequential Decision Makingcited by 8
  9. 2026ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulationcited by 2
  10. 2026EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learningcited by 2
  11. 2026Quo Vadis, World Modeling?cited by 0
  12. 2026Agent-Editing World Model: Rethinking World Modeling for LLM Agentscited by 0
2 more
  1. 2026Choosing Before Acting: Comparative Value Estimation for Long-Horizon Tool-Use Agentscited by 0
  2. 2026Software World Models: From Consequence Prediction to Decision Valuecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Gecko: A Simulation Environment with Stateful Feedback for Refining Agent Tool Calls
  2. 2024LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error
  3. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
  4. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments
  5. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks
  6. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs
  7. 2024Learning Evolving Tools for Large Language Models
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters