mlsanityData to 14 Oct 2026

Prompt a Robot to Walk with Large Language Models

arXiv 2309.09969 ↗First posted 18 Sep 2023cs.RO cs.LG cs.SYcited by 45

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) pre-trained on vast internet-scale data have showcased remarkable capabilities across diverse domains. Recently, there has been escalating interest in deploying LLMs for robotics, aiming to harness the power of foundation models in real-world settings. However, this approach faces significant challenges, particularly in grounding these models in the physical world and in generating dynamic robot motions. To address these issues, we introduce a novel paradigm in which we use few-shot prompts collected from the physical environment, enabling the LLM to autoregressively generate low-level control commands for robots without task-specific fine-tuning. Experiments across various robots and environments validate that our method can effectively prompt a robot to walk. We thus illustrate how LLMs can proficiently function as low-level feedback controllers for dynamic motion control even in high-dimensional robotic systems. The project website and source code can be found at: https://prompt2walk.github.io/ .

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 47 of its 58 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

47 of 58 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2017Proximal Policy Optimization Algorithmscited by 15,215
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
35 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  5. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  8. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  9. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  10. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  11. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  12. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  13. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  14. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  15. 2023Cognitive Architectures for Language Agentscited by 429
  16. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  17. 2023Larger language models do in-context learning differentlycited by 394
  18. 2022Visual Language Maps for Robot Navigationcited by 365
  19. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  20. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  21. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  22. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
  23. 2023Large Language Models as General Pattern Machinescited by 197
  24. 2022Rapid Locomotion via Reinforcement Learningcited by 168
  25. 2023Robot Parkour Learningcited by 149
  26. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 106
  27. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 71
  28. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 61
  29. 2021Fast and Efficient Locomotion via Learned Gait Transitionscited by 54
  30. 2022Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrationscited by 45
  31. 2023Barkour: Benchmarking Animal-level Agility with Quadruped Robotscited by 41
  32. 2023SayTap: Language to Quadrupedal Locomotioncited by 41
  33. 2021GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Modelcited by 25
  34. 2022MoCapAct: A Multi-Task Dataset for Simulated Humanoid Controlcited by 25
  35. 2022Learning Low-Frequency Motion Control for Robust and Dynamic Robot Locomotioncited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 26; 2025: 9; 2026: 6.

45 papers.

Sort:

By area:

  1. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  2. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  3. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 116
  4. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  5. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 102
  6. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
  7. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 70
  8. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  9. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priorscited by 54
  10. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  11. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 52
  12. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
33 more
  1. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  2. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  3. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 27
  4. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  5. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 23
  6. 2024Why Solving Multi-agent Path Finding with Large Language Model has not Succeeded Yetcited by 22
  7. 2024CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Modelscited by 22
  8. 2024Language Guided Skill Discoverycited by 16
  9. 2024RePrompt: Planning by Automatic Prompt Engineering for Large Language Models Agentscited by 16
  10. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  11. 2025Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysiscited by 9
  12. 2024VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actionscited by 8
  13. 2024ELEMENTAL: Interactive Learning from Demonstrations and Vision-Language Models for Reward Design in Roboticscited by 8
  14. 2024HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulationcited by 7
  15. 2024Using Large Language Models for Parametric Shape Optimizationcited by 7
  16. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  17. 2024EMOTION: Expressive Motion Sequence Generation for Humanoid Robots with In-Context Learningcited by 6
  18. 2024MaxMI: A Maximal Mutual Information Criterion for Manipulation Concept Discoverycited by 5
  19. 2024Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMscited by 4
  20. 2024A Large Language Model-based multi-agent manufacturing system for intelligent shopfloorcited by 4
  21. 2025Graphormer-Guided Task Planning: Beyond Static Rules with LLM Safety Perceptioncited by 4
  22. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  23. 2026Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agentscited by 4
  24. 2025A Status Quo Investigation of Large Language Models towards Cost-Effective CFD Automation with OpenFOAMGPT: ChatGPT vs. Qwen vs. Deepseekcited by 3
  25. 2024Training microrobots to swim by a large language modelcited by 2
  26. 2024Semantic Layering in Room Segmentation via LLMscited by 1
  27. 2025Semantic Intelligence: Integrating GPT-4 with A Planning in Low-Cost Roboticscited by 1
  28. 2025INTENTION: Inferring Tendencies of Humanoid Robot Motion Through Interactive Intuition and Grounded VLMcited by 1
  29. 2026PRISM-XR: Empowering Privacy-Aware XR Collaboration with Multimodal Large Language Modelscited by 1
  30. 2026TurboAgent: An LLM-Driven Autonomous Multi-Agent Framework for Turbomachinery Aerodynamic Designcited by 1
  31. 2026RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Modelscited by 0
  32. 2026Learning Task-Invariant Properties via Dreamer: Enabling Efficient Policy Transfer for Quadruped Robotscited by 0
  33. 2026PopGen: Generating Semantically Diverse Populations for Zero-Shot Coordinationcited by 0