Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence
On the map: Vision-language-action models → Failure recovery and robustness also fits: World model-based VLA
also in 1 more
Abstract (arXiv)
Vision-language-action (VLA) and world-action (WAM) models map observations and instructions directly to robot actions. This directness ties a policy to training: minor layout or viewpoint changes cause failure, and instructions generalize poorly. The root cause lies in representation: task requirements, conditions, progress, and failure recovery are implicitly encoded in action sequences, making them difficult to inspect or revise. Digital coding agents offer a precedent: LLMs call tools, verify results, and revise from feedback as executable code. The same working pattern of explicit state, manageable execution, and revisable procedures underlies generalization and long-horizon execution in the physical world, letting physical experience return as reusable programs, memory, or evidence. We propose Physical Coding, representing task state and execution as code. Code as World records objects, relations, constraints, and progress; Code as Policy organizes planning, verification, recovery, and execution. We build HexaAnything, which calls perception, planning, and control tools, including VLA/WAM policies, and makes in-the-loop decisions from external feedback. Verified traces become data and memory, enabling evolution from tools and Harness to model weights, architectures, and ultimately hardware and task design. On RoboCasa365, HexaAnything improves Composite-Unseen and overall success over XR-1 VLA, and its Harness-trained HexaModel beats the base on every split, indicating code traces internalize physical execution. On PhyBench and a dual-arm AgileX robot, the agent autonomously completes physics experiments and most tabletop tasks, often faster than published results. We observe data, model, and tool self-evolution; future work targets weight internalization, autonomous redesign of architectures, languages, representations, and tasks, and deployment in manufacturing and science.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models. Builds on 99 of its 109 references that are in the map. First posted on arXiv 28 Sep 2026.
Builds on
99 of 109 references.
Closest ancestors · 4
Same direction, newest first
- 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation23 Mar
- 2026LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries21 Jan
- 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models15 Oct
- 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization4 Oct
Other references · 95
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
38 more
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
- 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
- 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
- 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
- 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
- 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
- 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
- 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
- 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
- 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
- 2023Learning Interactive Real-World Simulatorscited by 440
- 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
- 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
- 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
- 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
- 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
- 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
- 2026Causal World Modeling for Robot Controlcited by 289
- 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
- 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
- 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
- 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
- 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 204
- 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
- 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
- 2025Do generative video models understand physical principles?cited by 151
- 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
- 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
- 2023Vision-Language Models as Success Detectorscited by 138
50 most cited of 95 All 95 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers
- 2026RACaP: Agentic Reasoning, Acting, and Coding as Policies for Evolvable Robot Learning
- 2026Encore: Few-Shot Agentic Discovery of Manipulation Strategies
- 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skills
- 2026Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation
- 2026EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents
- 2026RHO: Your Coding Agent is Secretly a Roboticist
- 2026Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures