Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools
Not on the map. Why?
Abstract (arXiv)
Frontier models can control robots, but reasoning through every reach, grasp, and retreat makes manipulation slow and token-intensive. We revisit code as policy with a different division of labor: models build executable tools, code handles multi-phase motions, and models decide what to do next. We introduce URAI (Universal Robot-Agent Interface), which couples a programming agent that constructs robot tools with an execution agent that uses them in a feedback loop. The programming agent writes reusable and task-specific tools from task intent and refines them through execution feedback and human guidance. The execution agent selects and parameterizes these tools from current observations; each call runs a complete motion locally before returning control to the agent. Unlike delegating subsequent decisions to a generated program, this design retains model-level decision-making between tool executions. Validated tool revisions persist across episodes without updating foundation-model weights, and a shared GUI and API make the same tools available to humans and agents. Across five RoboDojo tasks and four frozen execution agents, URAI raises aggregate success from 18.0% to 53.0% relative to direct fingertip control, with the largest gain on Swap Blocks; with the same tools, a program written in advance reaches only 24% against 56% for two agents deciding after each call. Three of the four agents also finish episodes 1.3-1.5 times faster with 1.5-1.7 times fewer execution-agent output tokens; DeepSeek-V4-Flash's cost barely changes. We further evaluate URAI on seven real-world AgileX dual-arm tasks, spanning object manipulation, cloth folding, and human-interactive tic-tac-toe. URAI connects the coding and decision-making capabilities of frontier agents, organizing robot control around reusable tools that agents can both invoke and revise.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
26 of 43 references.
References · 26
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
- 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
- 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
- 2023Large Language Models as Tool Makerscited by 303
- 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 289
- 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 133
14 more
- 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
- 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 129
- 2026RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policiescited by 40
- 2025OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agentscited by 38
- 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 20
- 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 20
- 2017A New Approach to Time-Optimal Path Parameterization based on Reachability Analysiscited by 14
- 2026An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyondcited by 11
- 2025Spatial Policy: Guiding Visuomotor Robotic Manipulation with Spatial-Aware Modeling and Reasoningcited by 10
- 2024What's the Move? Hybrid Imitation Learning via Salient Pointscited by 9
- 2026RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Roboticscited by 6
- 2026Transferring the Intelligence of VLMs to Robotic Controlcited by 4
- 2026InSight: Self-Guided Skill Acquisition via Steerable VLAscited by 2
- 2025Sketch Interface for Teleoperation of Mobile Manipulator to Enable Intuitive and Intended Operation: A Proof of Conceptcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026SEAR: Self-Evolving Agentic Roboticscited by 0
- 2026SEAR: Self-Evolving Agentic Robotics13 Oct 2026