# ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application

arXiv: https://arxiv.org/abs/2304.03893 · HTML: https://mlsanity.com/paper/2304.03893/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2023.

## Built on by (most cited first, 30 of 48)

- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers](https://mlsanity.com/paper/2306.06531/) (2023)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [Large Language Models for Robotics: Opportunities, Challenges, and Perspectives](https://mlsanity.com/paper/2401.04334/) (2024)
- [GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration](https://mlsanity.com/paper/2311.12015/) (2023)
- [Real-World Robot Applications of Foundation Models: A Review](https://mlsanity.com/paper/2402.05741/) (2024)
- [AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning](https://mlsanity.com/paper/2405.16247/) (2024)
- [TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage](https://mlsanity.com/paper/2308.03427/) (2023)
- [CoPAL: Corrective Planning of Robot Actions with Large Language Models](https://mlsanity.com/paper/2310.07263/) (2023)
- [TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems](https://mlsanity.com/paper/2311.11315/) (2023)
- [Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs](https://mlsanity.com/paper/2308.13812/) (2023)
- [LLM as A Robotic Brain: Unifying Egocentric Memory and Control](https://mlsanity.com/paper/2304.09349/) (2023)
- [HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments](https://mlsanity.com/paper/2401.12975/) (2024)
- [An Interactive Agent Foundation Model](https://mlsanity.com/paper/2402.05929/) (2024)
- [Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models](https://mlsanity.com/paper/2309.04316/) (2023)
- [AutoGPT+P: Affordance-based Task Planning with Large Language Models](https://mlsanity.com/paper/2402.10778/) (2024)
- [Decoding ChatGPT: A Taxonomy of Existing Research, Current Challenges, and Possible Future Directions](https://mlsanity.com/paper/2307.14107/) (2023)
- [InterAct: Exploring the Potentials of ChatGPT as a Cooperative Agent](https://mlsanity.com/paper/2308.01552/) (2023)
- [Learning Compositional Behaviors from Demonstration and Language](https://mlsanity.com/paper/2505.21981/) (2025)
- [A Safer Vision-based Autonomous Planning System for Quadrotor UAVs with Dynamic Obstacle Trajectory Prediction and Its Application with LLMs](https://mlsanity.com/paper/2311.12893/) (2023)
- [LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey](https://mlsanity.com/paper/2402.14558/) (2024)
- [RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models](https://mlsanity.com/paper/2403.06420/) (2024)
- [Code-as-Symbolic-Planner: Foundation Model-Based Robot Planning via Symbolic Code Generation](https://mlsanity.com/paper/2503.01700/) (2025)
- [Selective Perception: Optimizing State Descriptions with Reinforcement Learning for Language Model Actors](https://mlsanity.com/paper/2307.11922/) (2023)
- [Don't Let Your Robot be Harmful: Responsible Robotic Manipulation via Safety-as-Policy](https://mlsanity.com/paper/2411.18289/) (2024)
- [AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robots](https://mlsanity.com/paper/2409.11905/) (2024)
- [Structured Preference Optimization for Vision-Language Long-Horizon Task Planning](https://mlsanity.com/paper/2502.20742/) (2025)
- [WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model](https://mlsanity.com/paper/2308.15962/) (2023)
- [Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Models](https://mlsanity.com/paper/2408.07975/) (2024)
- [MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments](https://mlsanity.com/paper/2402.00290/) (2024)

## Abstract (arXiv)

This paper demonstrates how OpenAI's ChatGPT can be used in a few-shot setting to convert natural language instructions into a sequence of executable robot actions. The paper proposes easy-to-customize input prompts for ChatGPT that meet common requirements in practical applications, such as easy integration with robot execution systems and applicability to various environments while minimizing the impact of ChatGPT's token limit. The prompts encourage ChatGPT to output a sequence of predefined robot actions, represent the operating environment in a formalized style, and infer the updated state of the operating environment. Experiments confirmed that the proposed prompts enable ChatGPT to act according to requirements in various environments, and users can adjust ChatGPT's output with natural language feedback for safe and robust operation. The proposed prompts and source code are open-source and publicly available at https://github.com/microsoft/ChatGPT-Robot-Manipulation-Prompts