# RobotGPT: Robot Manipulation Learning from ChatGPT

arXiv: https://arxiv.org/abs/2312.01421 · HTML: https://mlsanity.com/paper/2312.01421/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2023.

## Built on by (most cited first, 30 of 39)

- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [Recover: A Neuro-Symbolic Framework for Failure Detection and Recovery](https://mlsanity.com/paper/2404.00756/) (2024)
- [Towards Open-World Grasping with Large Vision-Language Models](https://mlsanity.com/paper/2406.18722/) (2024)
- [P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Task](https://mlsanity.com/paper/2409.11279/) (2024)
- [Robot Collapse: Supply Chain Backdoor Attacks Against VLM-based Robotic Manipulation](https://mlsanity.com/paper/2411.11683/) (2024)
- [RoboDexVLM: Visual Language Model-Enabled Task Planning and Motion Control for Dexterous Robot Manipulation](https://mlsanity.com/paper/2503.01616/) (2025)
- [Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directions](https://mlsanity.com/paper/2502.15336/) (2025)
- [LLMs for Coding and Robotics Education](https://mlsanity.com/paper/2402.06116/) (2024)
- [Hierarchical Planning for Complex Tasks with Knowledge Graph-RAG and Symbolic Verification](https://mlsanity.com/paper/2504.04578/) (2025)
- [Grounding Actions in Camera Space: Observation-Centric Vision-Language-Action Policy](https://mlsanity.com/paper/2508.13103/) (2025)
- [Tell Me Where You Are: Multimodal LLMs Meet Place Recognition](https://mlsanity.com/paper/2406.17520/) (2024)
- [Survey of Vision-Language-Action Models for Embodied Manipulation](https://mlsanity.com/paper/2508.15201/) (2025)
- [VernaCopter: Disambiguated Natural-Language-Driven Robot via Formal Specifications](https://mlsanity.com/paper/2409.09536/) (2024)
- [TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models](https://mlsanity.com/paper/2412.15462/) (2024)
- [GenSwarm: Scalable Multi-Robot Code-Policy Generation and Deployment via Language Models](https://mlsanity.com/paper/2503.23875/) (2025)
- [Foundation Models for Autonomous Robots in Unstructured Environments](https://mlsanity.com/paper/2407.14296/) (2024)
- [HybridGen: VLM-Guided Hybrid Planning for Scalable Data Generation of Imitation Learning](https://mlsanity.com/paper/2503.13171/) (2025)
- [The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare](https://mlsanity.com/paper/2411.03287/) (2024)
- [Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation with Large Language Models](https://mlsanity.com/paper/2503.04280/) (2025)
- [OmniDexVLG: Learning Dexterous Grasp Generation from Vision Language Model-Guided Grasp Semantics, Taxonomy and Functional Affordance](https://mlsanity.com/paper/2512.03874/) (2025)
- [LLM-guided Task and Motion Planning using Knowledge-based Reasoning](https://mlsanity.com/paper/2412.07493/) (2024)
- [ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policy](https://mlsanity.com/paper/2502.09649/) (2025)
- [Boosting Robotic Manipulation Generalization with Minimal Costly Data](https://mlsanity.com/paper/2503.19516/) (2025)
- [RoboPilot: Generalizable Dynamic Robotic Manipulation with Dual-thinking Modes](https://mlsanity.com/paper/2510.00154/) (2025)
- [Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition](https://mlsanity.com/paper/2602.15124/) (2026)
- [SADP: Subgoal-Aware Diffusion Policy for Long-Horizon Manipulation Learned from Foundation Model Generated Demonstrations](https://mlsanity.com/paper/2605.16871/) (2026)
- [Human-centered In-building Embodied Delivery Benchmark](https://mlsanity.com/paper/2406.17898/) (2024)
- [D-RMGPT: Robot-assisted collaborative tasks driven by large multimodal models](https://mlsanity.com/paper/2408.11761/) (2024)
- [REFLEX: Metacognitive Reasoning for Reflective Zero-Shot Robotic Planning with Large Language Models](https://mlsanity.com/paper/2505.14899/) (2025)
- [LLM-Cave: A benchmark and light environment for large language models reasoning and decision-making system](https://mlsanity.com/paper/2511.22598/) (2025)

## Abstract (arXiv)

We present RobotGPT, an innovative decision framework for robotic manipulation that prioritizes stability and safety. The execution code generated by ChatGPT cannot guarantee the stability and safety of the system. ChatGPT may provide different answers for the same task, leading to unpredictability. This instability prevents the direct integration of ChatGPT into the robot manipulation loop. Although setting the temperature to 0 can generate more consistent outputs, it may cause ChatGPT to lose diversity and creativity. Our objective is to leverage ChatGPT's problem-solving capabilities in robot manipulation and train a reliable agent. The framework includes an effective prompt structure and a robust learning model. Additionally, we introduce a metric for measuring task difficulty to evaluate ChatGPT's performance in robot manipulation. Furthermore, we evaluate RobotGPT in both simulation and real-world environments. Compared to directly using ChatGPT to generate code, our framework significantly improves task success rates, with an average increase from 38.5% to 91.5%. Therefore, training a RobotGPT by utilizing ChatGPT as an expert is a more stable approach compared to directly using ChatGPT as a task planner.