mlsanityData to 7 Oct 2026

VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents

arXiv 2510.16907 ↗First posted 19 Oct 2025cs.AI cs.CLcited by 60

On the map: World models → World models for language agents

also in 3 more
Abstract (arXiv)

A key challenge in training Vision-Language Model (VLM) agents, compared to Language Model (LLM) agents, lies in the shift from textual states to complex visual observations. This transition introduces partial observability and demands robust world modeling. We ask: Can VLM agents construct internal world models through explicit visual state reasoning? To address this question, we architecturally enforce and reward the agent's reasoning process via reinforcement learning (RL), formulating it as a Partially Observable Markov Decision Process (POMDP). We find that decomposing the agent's reasoning into State Estimation ("what is the current state?") and Transition Modeling ("what comes next?") is critical for success, as demonstrated through five reasoning strategies. Our investigation into how agents represent internal beliefs reveals that the optimal representation is task-dependent: Natural Language excels at capturing semantic relationships in general tasks, while Structured formats are indispensable for precise manipulation and control. Building on these insights, we design a World Modeling Reward that provides dense, turn-level supervision for accurate state prediction, and introduce Bi-Level General Advantage Estimation (Bi-Level GAE) for turn-aware credit assignment. Through this form of visual state reasoning, a 3B-parameter model achieves a score of 0.82 across five diverse agent benchmarks, representing a 3× improvement over its untrained counterpart (0.21) and outperforming proprietary reasoning models such as GPT-5 (0.75), Gemini 2.5 Pro (0.67) and Claude 4.5 (0.62). All experiments are conducted within our VAGEN framework, a scalable system for training and analyzing multi-turn VLM agents in diverse visual environments. Code and data are publicly available at https://vagen-ai.github.io.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 47 of its 82 references that are in the map. First posted on arXiv 19 Oct 2025.

Builds on

47 of 82 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Critique of World Model7 Jul
  2. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learning8 Oct

Other references · 45

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2024A Survey on LLM-as-a-Judgecited by 1,384
33 more
  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  2. 2024OpenAI o1 System Cardcited by 1,220
  3. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  6. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  7. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  8. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  9. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  10. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  11. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  12. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  13. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  14. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  15. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  16. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  17. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  18. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 163
  19. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  20. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  21. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  22. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  23. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  24. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  25. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
  26. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  27. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  28. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Modelscited by 48
  29. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  30. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  31. 2024Gymnasium: A Standard Interface for Reinforcement Learning Environmentscited by 35
  32. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 24
  33. 2024ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Taskscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 53.

60 papers.

Sort:

By area:

  1. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
  2. 2026Qwen-AgentWorld: Language World Models for General Agentscited by 22
  3. 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 18
  4. 2026ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agentscited by 17
  5. 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMscited by 16
  6. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 16
  7. 2026Current Agents Fail to Leverage World Model as Tool for Foresightcited by 15
  8. 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interactioncited by 14
  9. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 14
  10. 2026PROGRESSLM: Towards Progress Reasoning in Vision-Language Modelscited by 13
  11. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 10
  12. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agentscited by 9
38 more
  1. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  2. 2025Abstract 3D Perception for Spatial Intelligence in Vision-Language Modelscited by 7
  3. 2026Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Modelscited by 7
  4. 2026Generalization in Online Reinforcement Learning for Mobile Agentscited by 6
  5. 2026OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agentscited by 6
  6. 2026Self-Improving World Modelling with Latent Actionscited by 5
  7. 2026AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agentscited by 5
  8. 2026Beyond Next-Observation Prediction: Agent-Authored World Modeling for Sequential Decision Makingcited by 5
  9. 2026MEDVISTAGYM: A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learningcited by 4
  10. 2026UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamicscited by 4
  11. 2026Nano World Models: A Minimalist Implementation of Future Video Predictioncited by 4
  12. 2026CoMAP: Co-Evolving World Models and Agent Policies for LLM Agentscited by 4
  13. 2026EnvRL: Learn from Environment Dynamics in Agentic Reinforcement Learningcited by 4
  14. 2025GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Trainingcited by 3
  15. 2026ProAct: Agentic Lookahead in Interactive Environmentscited by 3
  16. 2026Recursive Belief Vision Language Action Modelscited by 3
  17. 2026Gym-V: A Unified Vision Environment System for Agentic Vision Researchcited by 3
  18. 2026Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interactioncited by 3
  19. 2026ELITE: Experiential Learning and Intent-Aware Transfer for Self-improving Embodied Agentscited by 3
  20. 2026PhyWorld: Physics-Faithful World Model for Video Generationcited by 3
  21. 2026Hera: Learning Long-Horizon Coordination for Device-Cloud Collaborative LLM Agentscited by 3
  22. 2025Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interactioncited by 2
  23. 2025Active Intelligence in Video Avatars via Closed-loop World Modelingcited by 2
  24. 2026Reward Prediction with Factorized World Statescited by 2
  25. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learningcited by 2
  26. 2026Planning with the Viewscited by 2
  27. 2026CAST: Game Solvers as Turn-Level Teachers for LLM Agentscited by 2
  28. 2026Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learningcited by 2
  29. 2026Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoningcited by 2
  30. 2026Language-based Trial and Error Falls Behind in the Era of Experiencecited by 1
  31. 2026What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiositycited by 1
  32. 2026Efficient Agentic Reasoning Through Self-Regulated Simulative Planningcited by 1
  33. 2026HarnessBridge: Learnable Bidirectional Controller for LLM Agent Harnesscited by 1
  34. 2026Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agentscited by 1
  35. 2026PRACTICE: From Experience to Expertise in Self-Evolving Embodied Agentscited by 1
  36. 2026DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculumcited by 1
  37. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0
  38. 2026Seeing Isn't Believing: Mitigating Belief Inertia via Active Intervention in Embodied Agentscited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Making
  2. 2026Towards a Belief-Based World Model for LLM Agents
  3. 2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
  4. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
  5. 2026Insight-V++: Towards Advanced Long-Chain Visual Reasoning with Multimodal Large Language Models
  6. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning
  7. 2025RLVR-World: Training World Models with Reinforcement Learning
  8. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation