Slot-Level Robotic Placement via Visual Imitation from Single Human Video
Not on the map. Why?
Abstract (arXiv)
The majority of modern robot learning methods focus on learning a set of pre-defined tasks with limited or no generalization to new tasks. Extending the robot skillset to novel tasks involves gathering an extensive amount of training data for additional tasks. In this paper, we address the problem of teaching new tasks to robots using human demonstration videos for repetitive tasks (e.g., packing). This task requires understanding the human video to identify which object is being manipulated (the pick object) and where it is being placed (the placement slot). In addition, it needs to re-identify the pick object and the placement slots during inference along with the relative poses to enable robot execution of the task. To tackle this, we propose SLeRP, a modular system that leverages several advanced visual foundation models and a novel slot-level placement detector Slot-Net, eliminating the need for expensive video demonstrations for training. We evaluate our system using a new benchmark of real-world videos. The evaluation results show that SLeRP outperforms several baselines and can be deployed on a real robot.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 64 of its 82 references that are in the map. First posted on arXiv 2 Apr 2025.
Builds on
64 of 82 references.
References · 64
- 2023GPT-4 Technical Reportcited by 18,396
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Segment Anythingcited by 8,234
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2024GPT-4o System Cardcited by 4,588
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
- 2024Grounding Image Matching in 3D with MASt3Rcited by 983
- 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 979
38 more
- 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
- 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
- 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
- 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
- 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
- 2024Latent Action Pretraining from Videoscited by 321
- 2020Integrated Task and Motion Planningcited by 290
- 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
- 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 223
- 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 218
- 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
- 2023Any-point Trajectory Modeling for Policy Learningcited by 215
- 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
- 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
- 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
- 2022Human-to-Robot Imitation in the Wildcited by 161
- 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 148
- 2024Flow as the Cross-Domain Manipulation Interfacecited by 132
- 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
- 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 119
- 2022VideoDex: Learning Dexterity from Internet Videoscited by 106
- 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 89
- 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
- 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
- 2022Changer: Feature Interaction is What You Need for Change Detectioncited by 79
- 2020Transformers for One-Shot Visual Imitationcited by 78
- 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 61
- 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 59
- 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 52
- 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 52
- 2019Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassemblycited by 51
- 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
- 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 50
- 2024Matching Anything by Segmenting Anythingcited by 50
- 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 49
- 2022TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulationcited by 47
- 2022Equivariant Transporter Networkcited by 46
- 2020Object Rearrangement Using Learned Implicit Collision Functionscited by 43
50 most cited of 64 All 64 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.