mlsanityData to 7 Oct 2026

RAPID: Robot Agentic Programming from Demonstrations

arXiv 2609.30249 ↗First posted 24 Sep 2026cs.RO cs.AI cs.CVcited by 0

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 45 of its 64 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

45 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks6 Jul
  2. 2026ASPIRE: Agentic /Skills Discovery for Robotics30 Jun
  3. 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real World18 Jun
  4. 2026Playful Agentic Robot Learning17 Jun
  5. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation23 Mar
  6. 2026RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks12 Mar

Other references · 39

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  5. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2025SAM 3: Segment Anything with Conceptscited by 835
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  10. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  11. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 411
  12. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
27 more
  1. 2020Integrated Task and Motion Planningcited by 281
  2. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  3. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  4. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
  5. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  6. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  7. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  8. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  9. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  10. 2022Global Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-dynamic Contact Modelscited by 46
  11. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 46
  12. 2019Graph-Structured Visual Imitationcited by 43
  13. 2024DenseMatcher: Learning 3D Semantic Correspondence for Category-Level Manipulation from a Single Democited by 34
  14. 2023HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulationcited by 27
  15. 2022Learning to Grasp the Ungraspable with Emergent Extrinsic Dexteritycited by 25
  16. 2023Universal Visual Decomposer: Long-Horizon Manipulation Made Easycited by 23
  17. 2023One-shot Imitation Learning via Interaction Warpingcited by 16
  18. 2025Learning Compositional Behaviors from Demonstration and Languagecited by 15
  19. 2021Contact Mode Guided Motion Planning for Quasidynamic Dexterous Manipulation in 3Dcited by 13
  20. 2020Contact Mode Guided Sampling-Based Planning for Quasistatic Dexterous Manipulation in 2Dcited by 10
  21. 2023NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptorscited by 8
  22. 2024PRIME: Scaffolding Manipulation Tasks with Behavior Primitives for Data-Efficient Imitation Learningcited by 8
  23. 2024CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objectscited by 8
  24. 2024One-Shot Transfer of Long-Horizon Extrinsic Manipulation Through Contact Retargetingcited by 8
  25. 2021Contact-Implicit Trajectory Optimization for Dynamic Object Manipulationcited by 7
  26. 2024One-Shot Manipulation Strategy Learning by Making Contact Analogiescited by 6
  27. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Modelscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Encore: Few-Shot Agentic Discovery of Manipulation Strategies
  2. 2026CodeActionBench: Evaluating Agentic Code-as-Policy for Embodied Manipulation
  3. 2026Revisiting the "Push-T" Robot Manipulation Task with Agentic Robotics
  4. 2026EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics
  5. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
  6. 2026ManiSkillFormer: Demonstration-Free Compositional Manipulation via Geometric Contracts and Agentic Skill Graph
  7. 2026Real2Gym: Building Gyms from Videos, Bringing Skills to Robots
  8. 2026SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining