mlsanityData to 14 Oct 2026

Vision-based Manipulation from Single Human Video with Open-World Object Graphs

arXiv 2405.20321 ↗First posted 30 May 2024cs.RO cs.CV cs.LGcited by 50

Not on the map. Why?

Abstract (arXiv)

This work presents an object-centric approach to learning vision-based manipulation skills from human videos. We investigate the problem of robot manipulation via imitation in the open-world setting, where a robot learns to manipulate novel objects from a single video demonstration. We introduce ORION, an algorithm that tackles the problem by extracting an object-centric manipulation plan from a single RGB or RGB-D video and deriving a policy that conditions on the extracted plan. Our method enables the robot to learn from videos captured by daily mobile devices and to generalize the policies to deployment environments with varying visual backgrounds, camera angles, spatial layouts, and novel object instances. We systematically evaluate our method on both short-horizon and long-horizon tasks, using RGB-D and RGB-only demonstration videos. Across varied tasks and demonstration types (RGB-D / RGB), we observe an average success rate of 74.4%, demonstrating the efficacy of ORION in learning from a single human video in the open world. Additional materials can be found on our project website: https://ut-austin-rpl.github.io/ORION-release.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 64 of its 94 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

64 of 94 references.

References · 64

  1. 2023Segment Anythingcited by 8,234
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2018Open3D: A Modern Library for 3D Data Processingcited by 721
  5. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  6. 2023CoTracker: It is Better to Track Togethercited by 522
  7. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  8. 2018Behavioral Cloning from Observationcited by 492
  9. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 433
  10. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 429
  11. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  12. 2023Reconstructing Hands in 3D with Transformerscited by 306
38 more
  1. 2021Towards Open World Object Detectioncited by 297
  2. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 241
  3. 2017One-Shot Imitation Learningcited by 233
  4. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 223
  5. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  6. 2023Putting the Object Back into Video Object Segmentationcited by 188
  7. 2018Generative Adversarial Imitation from Observationcited by 184
  8. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  9. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  10. 2022Human-to-Robot Imitation in the Wildcited by 161
  11. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 156
  12. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 136
  13. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
  14. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 115
  15. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  16. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 93
  17. 2023RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitationcited by 90
  18. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  19. 20226-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmarkcited by 86
  20. 2020Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphscited by 83
  21. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 81
  22. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 78
  23. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 75
  24. 2023XSkill: Cross Embodiment Skill Discoverycited by 75
  25. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Roboticscited by 74
  26. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  27. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 67
  28. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 65
  29. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 61
  30. 2019StructureNet: Hierarchical Graph Networks for 3D Shape Generationcited by 59
  31. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 52
  32. 2023Zero-Shot Robot Manipulation from Passive Human Videoscited by 51
  33. 2017Deep Object-Centric Representations for Generalizable Robot Learningcited by 50
  34. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 46
  35. 2024DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Modelscited by 44
  36. 2018Deep Object-Centric Policies for Autonomous Drivingcited by 41
  37. 2021Tangent Space Backpropagation for 3D Transformation Groupscited by 40
  38. 2022Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learningcited by 38

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 23; 2026: 12.

50 papers.

Sort:

By area:

  1. 2024Latent Action Pretraining from Videoscited by 321
  2. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  3. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 145
  4. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 145
  5. 2024Flow as the Cross-Domain Manipulation Interfacecited by 132
  6. 2025FLARE: Robot Learning with Implicit World Modelingcited by 113
  7. 2025Humanoid Policy ~ Human Policycited by 78
  8. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 67
  9. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 59
  10. 2025VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulationcited by 46
  11. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  12. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 41
38 more
  1. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 38
  2. 2024AffordDP: Generalizable Diffusion Policy with Transferable Affordancecited by 37
  3. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 33
  4. 2024BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulationcited by 32
  5. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 32
  6. 2025X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Realcited by 28
  7. 2025ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstrationcited by 20
  8. 2025Robot Learning from a Physical World Modelcited by 20
  9. 2026SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulationcited by 15
  10. 2025MimicFunc: Imitating Tool Manipulation from a Single Human Video via Functional Correspondencecited by 13
  11. 2024One-Shot Imitation under Mismatched Executioncited by 12
  12. 2025FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulationcited by 10
  13. 2024One-Shot Manipulation Strategy Learning by Making Contact Analogiescited by 8
  14. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Modelscited by 7
  15. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  16. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrationscited by 6
  17. 2024TieBot: Learning to Knot a Tie from Visual Demonstration through a Real-to-Sim-to-Real Approachcited by 5
  18. 2026Robot Learning from Human Videos: A Surveycited by 5
  19. 2025RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrationscited by 4
  20. 2025Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generationcited by 4
  21. 2026Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Modelscited by 4
  22. 2024One-shot Video Imitation via Parameterized Symbolic Abstraction Graphscited by 3
  23. 2025Slot-Level Robotic Placement via Visual Imitation from Single Human Videocited by 3
  24. 2026SUGAR: A Scalable Human-Video-Driven Generalizable Humanoid Loco-Manipulation Learning Frameworkcited by 3
  25. 2025Learning Predictive Visuomotor Coordinationcited by 2
  26. 2025TeleOpBench: A Simulator-Centric Benchmark for Dual-Arm Dexterous Teleoperationcited by 2
  27. 2025Weakly-Supervised Learning of Dense Functional Correspondencescited by 2
  28. 2026EasyMimic: A Low-Cost Framework for Robot Imitation Learning from Human Videoscited by 2
  29. 2026RAPID: Robot Agentic Programming from Demonstrationscited by 2
  30. 2026KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videoscited by 1
  31. 2026In-Context Learning for Robots: Methods and Applicationscited by 1
  32. 2024Learning Generalizable 3D Manipulation With 10 Demonstrationscited by 0
  33. 2025Imitation Learning with Precisely Labeled Human Demonstrationscited by 0
  34. 2025Object-Focus Actor for Data-efficient Robot Generalization Dexterous Manipulationcited by 0
  35. 2026LIBERO-MAX: Do Robot Policies Adapt When the World Changes?cited by 0
  36. 2026Track-and-Complete: Learning Humanoid Skills from a Single Failed Human Videocited by 0
  37. 2026GOTT: Object-centric Dexterous Manipulation with a Reusable Cross-Embodiment Primitivecited by 0
  38. 2026Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Handcited by 0