World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal
On the map: LLM reasoning → Spatial and embodied reasoning
also in 2 more
Abstract (arXiv)
General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 39 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
31 of 39 references.
Closest ancestors · 3
Same direction, newest first
- 2025Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting26 Sep
- 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation13 May
- 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation3 Sep
Other references · 28
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
- 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
- 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
- 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
- 2026World Action Models are Zero-shot Policiescited by 411
- 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 311
- 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 274
- 2025Motus: A Unified Latent Action World Modelcited by 231
- 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 218
16 more
- 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
- 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
- 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
- 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 72
- 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraintscited by 58
- 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55
- 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 37
- 2024RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulationcited by 24
- 2026Playful Agentic Robot Learningcited by 18
- 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
- 2026MMSkills: Towards Multimodal Skills for General Visual Agentscited by 13
- 2026GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Taskscited by 11
- 2026Agent as Policy for Robotic Manipulationcited by 6
- 2026VIA: Visual Interface Agent for Robot Controlcited by 5
- 2026VISUALSKILL: Multimodal Skills for Computer-Use Agentscited by 3
- 2026AtomBridge: Agentic VLA Inference Plugin for Long-Horizon Tasks in Scientific Experimentscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026AR-WAM: A Visual-Conditioned Agent-Ready World Action Model for Robotic Manipulation
- 2026Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation
- 2026PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulation
- 2026Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Models
- 2026CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
- 2026World Pilot: Steering Vision-Language-Action Models with World-Action Priors
- 2025ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulation
- 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey