mlsanityData to 7 Oct 2026

CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

arXiv 2603.22435 ↗First posted 23 Mar 2026cs.RO cs.AIcited by 72

On the map: Vision-language-action models → Failure recovery and robustness

also in 1 more
Abstract (arXiv)

"Code-as-Policy" considers how executable code can complement data-intensive Vision-Language-Action (VLA) methods, yet their effectiveness as autonomous controllers for embodied manipulation remains underexplored. We present CaP-X, an open-access framework for systematically studying Code-as-Policy agents in robot manipulation. At its core is CaP-Gym, an interactive environment in which agents control robots by synthesizing and executing programs that compose perception and control primitives. Building on this foundation, CaP-Bench evaluates frontier language and vision-language models across varying levels of abstraction, interaction, and perceptual grounding. Across 12 models, CaP-Bench reveals a consistent trend: performance improves with human-crafted abstractions but degrades as these priors are removed, exposing a dependence on designer scaffolding. At the same time, we observe that this gap can be mitigated through scaling agentic test-time computation--through multi-turn interaction, structured execution feedback, visual differencing, automatic skill synthesis, and ensembled reasoning--substantially improves robustness even when agents operate over low-level primitives. These findings allow us to derive CaP-Agent0, a training-free framework that recovers human-level reliability on several manipulation tasks in simulation and on real embodiments. We further introduce CaP-RL, showing reinforcement learning with verifiable rewards improves success rates and transfers from sim2real with minimal gap. Together, CaP-X provides a principled, open-access platform for advancing embodied coding agents.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models. Builds on 62 of its 103 references that are in the map. First posted on arXiv 23 Mar 2026.

Builds on

62 of 103 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization4 Oct

Other references · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  5. 2016OpenAI Gymcited by 1,684
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  12. 2025SAM 3: Segment Anything with Conceptscited by 835
  13. 2022A Generalist Agentcited by 802
  14. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  15. 2024DeepSeek-V3 Technical Reportcited by 724
  16. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  17. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  18. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  19. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  20. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  21. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  22. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  23. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  24. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  25. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  26. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  27. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  28. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  29. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 140
  30. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
  31. 2025A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulationcited by 125
  32. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  33. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 111
  34. 2025Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfercited by 86
  35. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 83
  36. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  37. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  38. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 72.

72 papers.

Sort:

By area:

  1. 2026ASPIRE: Agentic /Skills Discovery for Roboticscited by 44
  2. 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 37
  3. 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real Worldcited by 31
  4. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 18
  5. 2026Playful Agentic Robot Learningcited by 18
  6. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  7. 2026What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agentscited by 9
  8. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  9. 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harnesscited by 7
  10. 2026Addressing the Orchestration Gap in Generalist Robots via Physical Agencycited by 7
  11. 2026Towards the Harness of Embodied Agentscited by 7
  12. 2026ABot-Claw: A Foundation for Persistent, Cooperative, and Self-Evolving Robotic Agentscited by 6
38 more
  1. 2026APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policiescited by 5
  2. 2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
  3. 2026ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AIcited by 5
  4. 2026VIA: Visual Interface Agent for Robot Controlcited by 5
  5. 2026PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Executioncited by 5
  6. 2026VLCP: Vision Language Control Policy Closed-Loop Code Replanning for Robot Manipulationcited by 5
  7. 2026Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memorycited by 4
  8. 2026HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harnesscited by 3
  9. 2026InSight: Self-Guided Skill Acquisition via Steerable VLAscited by 2
  10. 2026Sequential Planning via Anchored Robotic Keypointscited by 2
  11. 2026STeP: Signal Temporal Logic for Precise Specifications for Action Generation with Vision Language Modelscited by 2
  12. 2026EmbodiedSWE: Coding Agents for Long Horizon Dexterous Roboticscited by 2
  13. 2026Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokenscited by 2
  14. 2026Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Compositioncited by 1
  15. 2026A Few Words Go a Long Way: Language Guided Robot Policy Synthesiscited by 1
  16. 2026You Don't Need To Stay in The Loop: An Agentic Robotics Loop for Robot-Policy Improvementcited by 1
  17. 2026HarnessPAI: An Evolving Harness for Physical AIcited by 1
  18. 2026Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentationcited by 1
  19. 2026RACaP: Agentic Reasoning, Acting, and Coding as Policies for Evolvable Robot Learningcited by 1
  20. 2026World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsalcited by 1
  21. 2026Coding Agents for Generalized Task and Motion Planning Problemscited by 1
  22. 2026RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineerscited by 1
  23. 2026Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agentscited by 1
  24. 2026Bimanual Robot Manipulation via Multi-Agent In-Context Learningcited by 0
  25. 2026Octopus Protocol: One-Shot Hardware Discovery and Control for AI Agents via Infrastructure-as-Promptscited by 0
  26. 2026Flame3D: Zero-shot Compositional Reasoning of 3D Scenes with Agentic Language Modelscited by 0
  27. 2026MobileSAM2: Lightweight Segment Anything for Spatial Intelligencecited by 0
  28. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 0
  29. 2026PROBE: Manipulation-Grounded Visual Question Answering with VLM Agentscited by 0
  30. 2026Revisiting the "Push-T" Robot Manipulation Task with Agentic Roboticscited by 0
  31. 2026Visual General Intelligence: A White Papercited by 0
  32. 2026WetRobo: A Reproducible Robot Kit for Coding Agents in Biological Laboratoriescited by 0
  33. 2026Learning and Transferring Closed-Loop Robot Softwarecited by 0
  34. 2026Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulationcited by 0
  35. 2026From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Interventioncited by 0
  36. 2026Generalizing Manipulation Skills with a Local Coding Agentcited by 0
  37. 2026AdaHVLA: Adaptive Harnesses for Long-Horizon Vision-Language-Action Executioncited by 0
  38. 2026RAPID: Robot Agentic Programming from Demonstrationscited by 0

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation
  2. 2026Faster and Better? Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration
  3. 2026AR-WAM: A Visual-Conditioned Agent-Ready World Action Model for Robotic Manipulation
  4. 2026Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation
  5. 2026MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models
  6. 2025ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
  7. 2026Demonstration-Free Robotic Control via LLM Agents
  8. 2025RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulation