mlsanityData to 7 Oct 2026

VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Model

arXiv 2410.08792 ↗First posted 11 Oct 2024cs.RO cs.AI cs.CVcited by 36

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision Language Models (VLMs) have recently been adopted in robotics for their capability in common sense reasoning and generalizability. Existing work has applied VLMs to generate task and motion planning from natural language instructions and simulate training data for robot learning. In this work, we explore using VLM to interpret human demonstration videos and generate robot task planning. Our method integrates keyframe selection, visual perception, and VLM reasoning into a pipeline. We named it SeeDo because it enables the VLM to ''see'' human demonstrations and explain the corresponding plans to the robot for it to ''do''. To validate our approach, we collected a set of long-horizon human videos demonstrating pick-and-place tasks in three diverse categories and designed a set of metrics to comprehensively benchmark SeeDo against several baselines, including state-of-the-art video-input VLMs. The experiments demonstrate SeeDo's superior performance. We further deployed the generated task plans in both a simulation environment and on a real robot arm.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 91 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

58 of 91 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Robotic Control via Embodied Chain-of-Thought Reasoning11 Jul
  2. 2024CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environments22 Mar
  3. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration20 Nov
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  5. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
38 more
  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  7. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  9. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  10. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  12. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  13. 2023Learning Interactive Real-World Simulatorscited by 440
  14. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  15. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  16. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  17. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  18. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  19. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  20. 2017One-Shot Imitation Learningcited by 232
  21. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  22. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  23. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  24. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  25. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  26. 2023Video Language Planningcited by 160
  27. 2022Human-to-Robot Imitation in the Wildcited by 157
  28. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  29. 2023Structured World Models from Human Videoscited by 144
  30. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  31. 2024Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detectioncited by 122
  32. 2023On Bringing Robots Homecited by 101
  33. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)cited by 85
  34. 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 48
  35. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  36. 2019Learning One-Shot Imitation from Humans without Humanscited by 32
  37. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  38. 2024DITTO: Demonstration Imitation by Trajectory Transformationcited by 28

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 23; 2026: 11.

36 papers.

Sort:

By area:

  1. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  2. 2024CityWalker: Learning Embodied Urban Navigation from Web-Scale Videoscited by 40
  3. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37
  4. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Modelscited by 23
  5. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  6. 2025DexVLG: Dexterous Vision-Language-Grasp Model at Scalecited by 19
  7. 2026BPP: Long-Context Robot Imitation Learning by Focusing on Key History Framescited by 17
  8. 2025Skywork-R1V3 Technical Reportcited by 16
  9. 2025Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalizationcited by 16
  10. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  11. 2026AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perceptioncited by 11
  12. 2026Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulationcited by 8
24 more
  1. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Promptcited by 7
  2. 2025ExploreVLM: Closed-Loop Robot Exploration Task Planning with Vision-Language Modelscited by 7
  3. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactionscited by 6
  4. 2025PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulationcited by 4
  5. 2025PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agentscited by 4
  6. 2025HybridGen: VLM-Guided Hybrid Planning for Scalable Data Generation of Imitation Learningcited by 3
  7. 2025SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimizationcited by 3
  8. 2025Stitch-a-Demo: Video Demonstrations from Multistep Descriptionscited by 1
  9. 2025APEX: Empowering LLMs with Physics-Based Task Planning for Real-time Insightcited by 1
  10. 2025T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Modelscited by 1
  11. 2025Long-Horizon Visual Imitation Learning via Plan and Code Reflectioncited by 1
  12. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  13. 2026WatchAct: A Benchmark for Behavior-Grounded Robot Manipulationcited by 1
  14. 2025Search-Based Software Engineering and AI Foundation Models: Current Landscape and Future Roadmapcited by 0
  15. 2025CARMA: Context-Aware Situational Grounding of Human-Robot Group Interactions by Combining Vision-Language Models with Object and Action Recognitioncited by 0
  16. 2025EgoLoc: A Generalizable Solution for Temporal Interaction Localization in Egocentric Videoscited by 0
  17. 2025Automated Procedural Analysis via Video-Language Models for AI-assisted Nursing Skills Assessmentcited by 0
  18. 2026Bridging Speech, Emotion, and Motion: a VLM-based Multimodal Edge-deployable Framework for Humanoid Robotscited by 0
  19. 2026Spatially Grounded Long-Horizon Task Planning in the Wildcited by 0
  20. 2026Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoningcited by 0
  21. 2026MERGE: Guided Vision-Language Models for Multi-Actor Event Reasoning and Grounding in Human-Robot Interactioncited by 0
  22. 2026Build on Priors: Vision--Language--Guided Neuro-Symbolic Imitation Learning for Data-Efficient Real-World Robot Manipulationcited by 0
  23. 2026Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulationcited by 0
  24. 2026KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videoscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024What Matters in Building Vision-Language-Action Models for Generalist Robots
  2. 2024VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions
  3. 2025VisualActBench: Can VLMs See and Act like a Human?
  4. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation
  5. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  6. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
  7. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
  8. 2024Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks