mlsanityData to 7 Oct 2026

Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer

arXiv 2609.31770 ↗First posted 24 Sep 2026cs.RO cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

General-purpose multimodal agents can write robot-control programs, but repeated exploration and model-mediated action selection can make execution slow. We study how external body knowledge, successful experience, and executable skills improve an XLeRobot controlled by GPT-6-Astra in a simulated and a physical elevator-button task. In 30 fixed-start simulation trials, complete robot geometry and camera information reduce mean completion time by 57.4% relative to a baseline with only the common control interface and no prior experience; images with synchronized action and state records reduce it by 68.6% without additional body assets. In nine paired comparisons (18 trials) at starts displaced by 10-100 cm, experience recorded at the original start reduces mean time by 58-63% relative to no experience, demonstrating generalization to the tested new starting positions. During experience experiments, GPT-6-Astra spontaneously generates a short visual-feedback program. Researcher-refactored versions reduce mean local-task time by 29-31% in 27 simulation trials. Finally, 12 real-robot trials using operator-confirmed button contact demonstrate sim2real reuse: at a shared nominal start, simulation XML assets and simulation experience reduce mean time by 53.0% and 49.9%, respectively; real experience also transfers to two new starts. These results suggest a practical way to build general-purpose manipulation experiments around GPT-6-Astra: supply machine-readable body descriptions and synchronized demonstrations, and turn useful agent-generated feedback routines into reusable skills, while the agent adapts actions from current images. We release all task prompts, trial-level experimental data, and acquired skill implementations at https://github.com/hesd10/astra-robot-sim2real.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 20 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

14 of 20 references.

References · 14

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  8. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  12. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
2 more
  1. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  2. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156

Built on by

Not cited yet.