# Agent as Policy for Robotic Manipulation

arXiv: https://arxiv.org/abs/2609.12541 · HTML: https://mlsanity.com/paper/2609.12541/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 11 Sep 2026.

## Builds on (most cited first, 28 of 28)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [World Action Models are Zero-shot Policies](https://mlsanity.com/paper/2602.15922/) (2026)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [Unified Video Action Model](https://mlsanity.com/paper/2503.00200/) (2025)
- [RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis](https://mlsanity.com/paper/2402.16117/) (2024)
- [ASPIRE: Agentic /Skills Discovery for Robotics](https://mlsanity.com/paper/2607.00272/) (2026)
- [Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents](https://mlsanity.com/paper/2607.08448/) (2026)
- [RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks](https://mlsanity.com/paper/2603.11558/) (2026)
- [RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation](https://mlsanity.com/paper/2402.14623/) (2024)
- [AutoMate: Specialist and Generalist Assembly Policies over Diverse Geometries](https://mlsanity.com/paper/2407.08028/) (2024)
- [GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks](https://mlsanity.com/paper/2607.05369/) (2026)
- [Addressing the Orchestration Gap in Generalist Robots via Physical Agency](https://mlsanity.com/paper/2607.21725/) (2026)

## Built on by (most cited first, 8 of 8)

- [An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond](https://mlsanity.com/paper/2609.24170/) (2026)
- [Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulation](https://mlsanity.com/paper/2609.20822/) (2026)
- [World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal](https://mlsanity.com/paper/2609.29964/) (2026)
- [SimEX: Simulation-Integrated Robotics AutoResearch](https://mlsanity.com/paper/2609.38982/) (2026)
- [Real2Gym: Building Gyms from Videos, Bringing Skills to Robots](https://mlsanity.com/paper/2609.37089/) (2026)
- [CAPEX: Efficiently Distilling Foundation Model Behavior into Deployable Robot Policies through Experience-Adaptive Reasoning](https://mlsanity.com/paper/2609.33007/) (2026)
- [Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies](https://mlsanity.com/paper/2610.06235/) (2026)
- [ChunkAct: Action Chunking for Agentic Robotics](https://mlsanity.com/paper/2610.18874/) (2026)

## Abstract (arXiv)

We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. AGP achieves success rates of at least 80% in seven of eight task configurations and significantly outperforms previous agentic robot systems. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.