mlsanityData to 7 Oct 2026

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

arXiv 2401.04334 ↗First posted 9 Jan 2024cs.RO cs.AIcited by 88

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large language models (LLMs) have undergone significant expansion and have been increasingly integrated across various domains. Notably, in the realm of robot task planning, LLMs harness their advanced reasoning and language comprehension capabilities to formulate precise and efficient action plans based on natural language instructions. However, for embodied tasks, where robots interact with complex environments, text-only LLMs often face challenges due to a lack of compatibility with robotic visual perception. This study provides a comprehensive overview of the emerging integration of LLMs and multimodal LLMs into various robotic tasks. Additionally, we propose a framework that utilizes multimodal GPT-4V to enhance embodied task planning through the combination of natural language instructions and robot visual perceptions. Our results, based on diverse datasets, indicate that GPT-4V effectively enhances robot performance in embodied tasks. This extensive survey and evaluation of LLMs and multimodal LLMs across a variety of robotic tasks enriches the understanding of LLM-centric embodied intelligence and provides forward-looking insights toward bridging the gap in Human-Robot-Environment interaction.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 93 of its 118 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

93 of 118 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  2. 2023Towards A Unified Agent with Foundation Models18 Jul
  3. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  4. 2022Visual Language Maps for Robot Navigation11 Oct

Other references · 89

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
38 more
  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  5. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  6. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  9. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  10. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  11. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  12. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  13. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  14. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  15. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  16. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  17. 2023Challenges and Applications of Large Language Modelscited by 251
  18. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  19. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  20. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  21. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  22. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  23. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  24. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  25. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  26. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  27. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  28. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  29. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  30. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  31. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  32. 2017Natural Language Processing: State of The Art, Current Trends and Challengescited by 114
  33. 2021Medical Visual Question Answering: A Surveycited by 93
  34. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  35. 2022MetaMorph: Learning Universal Controllers with Transformerscited by 78
  36. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  37. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  38. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 34; 2026: 24.

88 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2024Revolutionizing Finance with LLMs: An Overview of Applications and Insightscited by 88
  3. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  4. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  5. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  6. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  7. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  8. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  9. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  10. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
  11. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  12. 2024CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Masterycited by 30
38 more
  1. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMscited by 26
  2. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  3. 2024Large Language Models for Manufacturingcited by 24
  4. 2024A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddingscited by 23
  5. 2025PDE-Controller: LLMs for Autoformalization and Reasoning of PDEscited by 23
  6. 2024MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation modelscited by 22
  7. 2024IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor datacited by 18
  8. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  9. 2025Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robotscited by 17
  10. 2025Deception in LLMs: Self-Preservation and Autonomous Goals in Large Language Modelscited by 15
  11. 2024Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translationcited by 12
  12. 2024GP-GPT: Large Language Model for Gene-Phenotype Mappingcited by 10
  13. 2025VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carrierscited by 10
  14. 2024LLMs for Coding and Robotics Educationcited by 9
  15. 2024MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planningcited by 8
  16. 2024LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulationcited by 8
  17. 2025Large Language Model Based Multi-Agent System Augmented Complex Event Processing Pipeline for Internet of Multimedia Thingscited by 8
  18. 2026STAC: Plug-and-Play Spatio-Temporal Aware Cache Compression for Streaming 3D Reconstructioncited by 8
  19. 2024Details Make a Difference: Object State-Sensitive Neurorobotic Task Planningcited by 7
  20. 2024From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AIcited by 7
  21. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluationcited by 7
  22. 2026OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactionscited by 6
  23. 2024Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learningcited by 5
  24. 2024Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagerycited by 4
  25. 2025Brain-Inspired Exploration of Functional Networks and Key Neurons in Large Language Modelscited by 4
  26. 2025A Unified Framework for Real-Time Failure Handling in Robotics Using Vision-Language Models, Reactive Planner and Behavior Treescited by 4
  27. 2025Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planningcited by 4
  28. 2025Evaluating Language Models' Evaluations of Gamescited by 4
  29. 2025How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptationcited by 4
  30. 2024LLMSat: A Large Language Model-Based Goal-Oriented Agent for Autonomous Space Explorationcited by 3
  31. 2025Large Language Models for Bioinformaticscited by 3
  32. 2025MINDSTORES: Memory-Informed Neural Decision Synthesis for Task-Oriented Reinforcement in Embodied Systemscited by 3
  33. 2025LLM-Mediated Guidance of MARL Systemscited by 3
  34. 2025A Human-in-the-loop Approach to Robot Action Replanning through LLM Common-Sense Reasoningcited by 3
  35. 2025A Hierarchical Agentic Framework for Autonomous Drone-Based Visual Inspectioncited by 3
  36. 2026MOTIF: Learning Action Motifs for Few-shot Cross-Embodiment Transfercited by 3
  37. 2024A Comparison of Prompt Engineering Techniques for Task Planning and Execution in Service Roboticscited by 2
  38. 2024The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcarecited by 2

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
  2. 2025Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy
  3. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks
  4. 2024Robots Can Multitask Too: Integrating a Memory Architecture and LLMs for Enhanced Cross-Task Robot Action Generation
  5. 2025Large Language Models for Multi-Robot Systems: A Survey
  6. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications
  7. 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces
  8. 2025Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models