mlsanityData to 7 Oct 2026

GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration

arXiv 2311.12015 ↗First posted 20 Nov 2023cs.RO cs.CL cs.CVcited by 76

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We introduce a pipeline that enhances a general-purpose Vision Language Model, GPT-4V(ision), to facilitate one-shot visual teaching for robotic manipulation. This system analyzes videos of humans performing tasks and outputs executable robot programs that incorporate insights into affordances. The process begins with GPT-4V analyzing the videos to obtain textual explanations of environmental and action details. A GPT-4-based task planner then encodes these details into a symbolic task plan. Subsequently, vision systems spatially and temporally ground the task plan in the videos. Objects are identified using an open-vocabulary object detector, and hand-object interactions are analyzed to pinpoint moments of grasping and releasing. This spatiotemporal grounding allows for the gathering of affordance information (e.g., grasp types, waypoints, and body postures) critical for robot execution. Experiments across various scenarios demonstrate the method's efficacy in enabling real robots to operate from one-shot human demonstrations. Meanwhile, quantitative tests have revealed instances of hallucination in GPT-4V, highlighting the importance of incorporating human supervision within the pipeline. The prompts of GPT-4V/GPT-4 are available at this project page: https://microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts/

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 67 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

51 of 67 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Vision-Language Foundation Models as Effective Robot Imitators2 Nov
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 49

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  9. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  10. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
37 more
  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  2. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  3. 2020Integrated Task and Motion Planningcited by 281
  4. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  5. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  6. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  7. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 140
  8. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  9. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  10. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  11. 2023ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Applicationcited by 48
  12. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  13. 2023Creative Robot Tool Use with Large Language Modelscited by 42
  14. 2023Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specificationcited by 38
  15. 2023Interactive Task Planning with Language Modelscited by 36
  16. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  17. 2022CAPE: Corrective Actions from Precondition Errors using Large Language Modelscited by 30
  18. 2023GRID: Scene-Graph-based Instruction-driven Robotic Task Planningcited by 27
  19. 2023Integrating Action Knowledge and LLMs for Task Planning and Situation Handling in Open Worldscited by 26
  20. 2023Generalizable Long-Horizon Manipulations with Large Language Modelscited by 16
  21. 2023Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuningcited by 15
  22. 2022Instruction-Following Agents with Multimodal Transformercited by 13
  23. 2020DeComplex: Task planning from complex natural instructions by a collocating robotcited by 10
  24. 2022Robot Task Planning and Situation Handling in Open Worldscited by 10
  25. 2022Learning Neuro-symbolic Programs for Language Guided Robot Manipulationcited by 9
  26. 2023ERRA: An Embodied Representation and Reasoning Architecture for Long-horizon Language-conditioned Manipulation Taskscited by 8
  27. 2020A Learning-from-Observation Framework: One-Shot Robot Teaching for Grasp-Manipulation-Release Household Operationscited by 7
  28. 2020Translating Natural Language Instructions to Computer Programs for Robot Manipulationcited by 5
  29. 2021Semantic constraints to represent common sense required in household actions for multi-modal Learning-from-observation robotcited by 5
  30. 2022Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placementcited by 5
  31. 2020Verbal Focus-of-Attention System for Learning-from-Observationcited by 3
  32. 2020Task-oriented Motion Mapping on Robots of Various Configuration using Body Role Divisioncited by 3
  33. 2023Task-sequencing Simulator: Integrated Machine Learning to Execution Simulation for Robot Manipulationcited by 3
  34. 2023Applying Learning-from-observation to household service robots: three common-sense formulationcited by 3
  35. 2022Learning-from-Observation System Considering Hardware-Level Reusabilitycited by 2
  36. 2021Text-driven object affordance for guiding grasp-type recognition in multimodal robot teachingcited by 1
  37. 2022Interactive Task Encoding System for Learning-from-Observationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 37; 2025: 21; 2026: 17.

76 papers.

Sort:

By area:

  1. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  2. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  3. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  4. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  5. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  6. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  7. 2024Human-Object Interaction from Human-Level Instructionscited by 62
  8. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  9. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  10. 2024Dense Connector for MLLMscited by 48
  11. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  12. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
38 more
  1. 2024CityWalker: Learning Embodied Urban Navigation from Web-Scale Videoscited by 40
  2. 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulationcited by 40
  3. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 36
  4. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 33
  5. 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 32
  6. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  7. 2024IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Modelscited by 29
  8. 2024VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Contextcited by 29
  9. 2024ReplanVLM: Replanning Robotic Tasks with Visual Language Modelscited by 28
  10. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  11. 2025AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulationcited by 21
  12. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  13. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planningcited by 14
  14. 2024Towards Open-World Grasping with Large Vision-Language Modelscited by 13
  15. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 12
  16. 2024What Makes for Good Image Captions?cited by 11
  17. 2024Never-Ending Behavior-Cloning Agent for Robotic Manipulationcited by 10
  18. 2024Don't Let Your Robot be Harmful: Responsible Robotic Manipulation via Safety-as-Policycited by 9
  19. 2025Multi-Agent Systems for Robotic Autonomy with LLMscited by 9
  20. 2024MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planningcited by 8
  21. 2024VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actionscited by 8
  22. 2025REMAC: Self-Reflective and Self-Evolving Multi-Agent Collaboration for Long-Horizon Robot Manipulationcited by 8
  23. 2025Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulationcited by 8
  24. 2026SG-VLA: Learning Spatially-Grounded Vision-Language-Action Models for Mobile Manipulationcited by 8
  25. 2024GameVLM: A Decision-making Framework for Robotic Task Planning Based on Visual Language Models and Zero-sum Gamescited by 7
  26. 2024Details Make a Difference: Object State-Sensitive Neurorobotic Task Planningcited by 7
  27. 2024HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Modelscited by 7
  28. 2024Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigationcited by 7
  29. 2026Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulationcited by 7
  30. 2023Image and Data Mining in Reticular Chemistry Using GPT-4Vcited by 6
  31. 2024REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Modelscited by 6
  32. 2024Open-Vocabulary Action Localization with Iterative Visual Promptingcited by 5
  33. 2025Lemmanaid: Neuro-Symbolic Lemma Conjecturingcited by 5
  34. 2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
  35. 2025Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generationcited by 4
  36. 2024CLIPSwarm: Generating Drone Shows from Text Prompts with Vision-Language Modelscited by 3
  37. 2025Efficient Alignment of Unconditioned Action Prior for Language-conditioned Pick and Place in Cluttercited by 3
  38. 2026ROSClaw: An OpenClaw ROS 2 Framework for Agentic Robot Control and Interactioncited by 3

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
  2. 2026Embodied GPT-5.1: Evidence of a World Model?
  3. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks
  4. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
  5. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
  6. 2024Vision-Language Model-based Physical Reasoning for Robot Liquid Perception
  7. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting
  8. 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance