Robot Manipulation with GPT-6-Astra: Body Knowledge, Experience Reuse, Emergent Skills, and Sim2Real Transfer
Not on the map. Why?
Abstract (arXiv)
General-purpose multimodal agents can write robot-control programs, but repeated exploration and model-mediated action selection can make execution slow. We study how external body knowledge, successful experience, and executable skills improve an XLeRobot controlled by GPT-6-Astra in a simulated and a physical elevator-button task. In 30 fixed-start simulation trials, complete robot geometry and camera information reduce mean completion time by 57.4% relative to a baseline with only the common control interface and no prior experience; images with synchronized action and state records reduce it by 68.6% without additional body assets. In nine paired comparisons (18 trials) at starts displaced by 10-100 cm, experience recorded at the original start reduces mean time by 58-63% relative to no experience, demonstrating generalization to the tested new starting positions. During experience experiments, GPT-6-Astra spontaneously generates a short visual-feedback program. Researcher-refactored versions reduce mean local-task time by 29-31% in 27 simulation trials. Finally, 12 real-robot trials using operator-confirmed button contact demonstrate sim2real reuse: at a shared nominal start, simulation XML assets and simulation experience reduce mean time by 53.0% and 49.9%, respectively; real experience also transfers to two new starts. These results suggest a practical way to build general-purpose manipulation experiments around GPT-6-Astra: supply machine-readable body descriptions and synchronized demonstrations, and turn useful agent-generated feedback routines into reusable skills, while the agent adapts actions from current images. We release all task prompts, trial-level experimental data, and acquired skill implementations at https://github.com/hesd10/astra-robot-sim2real.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 20 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
14 of 20 references.
References · 14
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
- 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
- 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
2 more
Built on by
Not cited yet.