mlsanityData to 14 Oct 2026

ChatGPT for Robotics: Design Principles and Model Abilities

arXiv 2306.17582 ↗First posted 20 Feb 2023cs.AI cs.CL cs.HCcited by 299

Not on the map. Why?

Cited by 65 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents an experimental study regarding the use of OpenAI's ChatGPT for robotics applications. We outline a strategy that combines design principles for prompt engineering and the creation of a high-level function library which allows ChatGPT to adapt to different robotics tasks, simulators, and form factors. We focus our evaluations on the effectiveness of different prompt engineering techniques and dialog strategies towards the execution of various types of robotics tasks. We explore ChatGPT's ability to use free-form dialog, parse XML tags, and to synthesize code, in addition to the use of task-specific prompting functions and closed-loop reasoning through dialogues. Our study encompasses a range of tasks within the robotics domain, from basic logical, geometrical, and mathematical reasoning all the way to complex domains such as aerial navigation, manipulation, and embodied agents. We show that ChatGPT can be effective at solving several of such tasks, while allowing users to interact with it primarily via natural language instructions. In addition to these studies, we introduce an open-sourced research tool called PromptCraft, which contains a platform where researchers can collaboratively upload and vote on examples of good prompting schemes for robotics applications, as well as a sample robotics simulator with ChatGPT integration, making it easier for users to get started with using ChatGPT for robotics.

Key facts. Cited by 299 papers in the map (data through 14 Oct 2026). Builds on 34 of its 64 references that are in the map. First posted on arXiv 20 Feb 2023.

Builds on

34 of 64 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
22 more
  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  2. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  5. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  6. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  7. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  8. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 523
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  11. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  12. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  13. 2020Transformer Networks for Trajectory Forecastingcited by 181
  14. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  15. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  16. 2022Correcting Robot Plans with Natural Language Feedbackcited by 69
  17. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  18. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  19. 2022PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Trainingcited by 24
  20. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformerscited by 17
  21. 2021Zero-shot Task Adaptation using Natural Languagecited by 15
  22. 2021Language Understanding for Field and Service Robots in a Priori Unknown Environmentscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 103; 2024: 100; 2025: 57; 2026: 39.

299 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  6. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  7. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 490
  8. 2024Automated Design of Agentic Systemscited by 407
  9. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  10. 2023Tool Learning with Foundation Modelscited by 302
  11. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  12. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
38 more
  1. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  3. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  4. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 241
  5. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  6. 2023UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wildcited by 200
  7. 2023OpenAGI: When LLM Meets Domain Expertscited by 197
  8. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  9. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  10. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  11. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  12. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  13. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 160
  14. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 153
  15. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
  16. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  17. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  18. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 111
  19. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  20. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  21. 2024Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodologycited by 95
  22. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 93
  23. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 89
  24. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 87
  25. 2023Embodied Task Planning with Large Language Modelscited by 84
  26. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 79
  27. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 77
  28. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Roboticscited by 74
  29. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  30. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 71
  31. 2023Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agentscited by 69
  32. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 67
  33. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 67
  34. 2023RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Modelscited by 65
  35. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  36. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
  37. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  38. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61

50 most cited of 299 All 299 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Assessing the Impact of Prompting Methods on ChatGPT's Mathematical Capabilities
  2. 2024Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs
  3. 2022Interactive Language: Talking to Robots in Real Time
  4. 2023Exploring Large Language Models to Facilitate Variable Autonomy for Human-Robot Teaming
  5. 2023Chatbots as Problem Solvers: Playing Twenty Questions with Role Reversals
  6. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language
  7. 2023Improving ChatGPT Prompt for Code Generation
  8. 2025Exploring GPT-4 for Robotic Agent Strategy with Real-Time State Feedback and a Reactive Behaviour Framework