mlsanityData to 7 Oct 2026

Language Models as Zero-Shot Trajectory Generators

arXiv 2310.11604 ↗First posted 17 Oct 2023cs.RO cs.AI cs.CLcited by 59

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have recently shown promise as high-level planners for robots when given access to a selection of low-level skills. However, it is often assumed that LLMs do not possess sufficient knowledge to be used for the low-level trajectories themselves. In this work, we address this assumption thoroughly, and investigate if an LLM (GPT-4) can directly predict a dense sequence of end-effector poses for manipulation tasks, when given access to only object detection and segmentation vision models. We designed a single, task-agnostic prompt, without any in-context examples, motion primitives, or external trajectory optimisers. Then we studied how well it can perform across 30 real-world language-based tasks, such as "open the bottle cap" and "wipe the plate with the sponge", and we investigated which design choices in this prompt are the most important. Our conclusions raise the assumed limit of LLMs for robotics, and we reveal for the first time that LLMs do indeed possess an understanding of low-level robot control sufficient for a range of common tasks, and that they can additionally detect failures and then re-plan trajectories accordingly. Videos, prompts, and code are available at: https://www.robot-learning.uk/language-models-trajectory-generators.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

34 of 44 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Segment Anythingcited by 8,158
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2023A Survey of Large Language Modelscited by 2,537
22 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  5. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  9. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  11. 2023LERF: Language Embedded Radiance Fieldscited by 538
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  13. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  14. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  15. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  16. 2023Large Language Models as General Pattern Machinescited by 196
  17. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  18. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  19. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  20. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  21. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  22. 2023Towards A Unified Agent with Foundation Modelscited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 16; 2025: 25; 2026: 17.

59 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2023Large Language Models for Robotics: A Surveycited by 132
  3. 2024Vision Language Models are In-Context Value Learnerscited by 84
  4. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  5. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Roboticscited by 71
  6. 2024Comp4D: LLM-Guided Compositional 4D Scene Generationcited by 50
  7. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  8. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 36
  9. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  10. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 31
  11. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  12. 2024Local Policies Enable Zero-shot Long-horizon Manipulationcited by 21
38 more
  1. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulationcited by 18
  2. 2024Language Guided Skill Discoverycited by 16
  3. 2024Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robotscited by 14
  4. 2025Action Tokenizer Matters in In-Context Imitation Learningcited by 12
  5. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  6. 2025Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robotscited by 8
  7. 2026An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyondcited by 8
  8. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 7
  9. 2026Future Optical Flow Prediction Improves Robot Control & Video Generationcited by 7
  10. 2025ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Taskscited by 6
  11. 2024Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMscited by 4
  12. 2024TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Modelscited by 4
  13. 2025LuciBot: Automated Robot Policy Learning from Generated Videoscited by 4
  14. 2025IMPACT: Intelligent Motion Planning with Acceptable Contact Trajectories via Vision-Language Modelscited by 4
  15. 2025OpenNav: Open-World Navigation with Multimodal Large Language Modelscited by 3
  16. 2025GraspCorrect: Robotic Grasp Correction via Vision-Language Model-Guided Feedbackcited by 2
  17. 2025LLM-based Interactive Imitation Learning for Robotic Manipulationcited by 2
  18. 2025Automated Hybrid Reward Scheduling via Large Language Models for Robotic Skill Learningcited by 2
  19. 2025Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)cited by 2
  20. 2026SAIL: Test-Time Scaling for In-Context Imitation Learning with VLMcited by 2
  21. 2026How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Studycited by 2
  22. 2024Geometric Algebra Meets Large Language Models: Instruction-Based Transformations of Separate Meshes in 3D, Interactive and Controllable Scenescited by 1
  23. 2025Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation with Large Language Modelscited by 1
  24. 2025Trajectory Adaptation using Large Language Modelscited by 1
  25. 2025LLMSynthor: Macro-Aligned Micro-Records Synthesis with Large Language Modelscited by 1
  26. 2025LA-RCS: LLM-Agent-Based Robot Control Systemcited by 1
  27. 2025A 1000× Faster LLM-enhanced Algorithm For Path Planning in Large-scale Grid Mapscited by 1
  28. 2025MaP-AVR: A Meta-Action Planner for Agents Leveraging Vision Language Models and Retrieval-Augmented Generationcited by 1
  29. 2026Selective Perception for Robot: Task-Aware Attention in Multimodal VLAcited by 1
  30. 2026MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulationcited by 1
  31. 2026Toward Generalist Neural Motion Planners for Robotic Manipulators: Challenges and Opportunitiescited by 1
  32. 2026ThermoAct:Thermal-Aware Vision-Language-Action Models for Robotic Perception and Decision-Makingcited by 1
  33. 2026AMATA: Adaptive Multi-Agent Trajectory Alignment for Knowledge-Intensive Question Answeringcited by 1
  34. 2024Growing from Exploration: A self-exploring framework for robots based on foundation modelscited by 0
  35. 2024Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Modelscited by 0
  36. 2025Learning Model Successorscited by 0
  37. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0
  38. 2025Strategic Fusion of Vision Language Models: Shapley-Credited Context-Aware Dawid-Skene for Multi-Label Tasks in Autonomous Drivingcited by 0

50 most cited of 59 All 59 (JSON)