mlsanityData to 7 Oct 2026

Learning Human-to-Robot Handovers from Point Clouds

arXiv 2303.17592 ↗First posted 30 Mar 2023cs.RO cs.CV cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

We propose the first framework to learn control policies for vision-based human-to-robot handovers, a critical task for human-robot interaction. While research in Embodied AI has made significant progress in training robot agents in simulated environments, interacting with humans remains challenging due to the difficulties of simulating humans. Fortunately, recent research has developed realistic simulated environments for human-to-robot handovers. Leveraging this result, we introduce a method that is trained with a human-in-the-loop via a two-stage teacher-student framework that uses motion and grasp planning, reinforcement learning, and self-supervision. We show significant performance gains over baselines on a simulation benchmark, sim-to-sim transfer and sim-to-real transfer.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 45 of its 56 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

45 of 56 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  8. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  9. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  10. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  11. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  12. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
33 more
  1. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  2. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  3. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 244
  4. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 237
  5. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  6. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  7. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  8. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  9. 2021A System for General In-Hand Object Re-Orientationcited by 155
  10. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 151
  11. 2021Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Timecited by 140
  12. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 131
  13. 2018Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approachcited by 117
  14. 2020ACRONYM: A Large-Scale Grasp Dataset Based on Simulationcited by 115
  15. 2013Data-Driven Grasp Synthesis - A Surveycited by 104
  16. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  17. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  18. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  19. 2022Retrospectives on the Embodied AI Workshopcited by 46
  20. 2019Assistive Gym: A Physics Simulation Framework for Assistive Roboticscited by 44
  21. 2019A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Setcited by 30
  22. 2020Object Handovers: a Review for Roboticscited by 28
  23. 2020Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learningcited by 27
  24. 2020Reactive Human-to-Robot Handovers of Arbitrary Objectscited by 25
  25. 2020Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Cloudscited by 24
  26. 2020Object-Independent Human-to-Robot Handovers using Real Time Robotic Visioncited by 17
  27. 2021H2O: A Benchmark for Visual Human-human Object Handover Analysiscited by 16
  28. 2019Manipulation Trajectory Optimization with Online Grasp Synthesis and Selectioncited by 15
  29. 2021Dynamic Grasping with Reachability and Motion Awarenesscited by 15
  30. 2022HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handoverscited by 15
  31. 2021Towards safe human-to-robot handovers of unknown containerscited by 11
  32. 2022Model Predictive Control for Fluid Human-to-Robot Handoverscited by 9
  33. 2019Demonstration-Guided Deep Reinforcement Learning of Control Policies for Dexterous Human-Robot Interactioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 11; 2026: 4.

34 papers.

Sort:

By area:

  1. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  2. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 93
  3. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 88
  4. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 85
  5. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 80
  6. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 62
  7. 2023ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulationcited by 53
  8. 2024Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learningcited by 45
  9. 2024CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangementcited by 40
  10. 2024Neural MP: A Generalist Neural Motion Plannercited by 18
  11. 2024GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation, Demonstration, and Imitationcited by 14
  12. 2023SynH2R: Synthesizing Hand-Object Motions for Learning Human-to-Robot Handoverscited by 13
22 more
  1. 2024SyncDiff: Synchronized Motion Diffusion for Multi-Body Human-Object Interaction Synthesiscited by 12
  2. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  3. 2024RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Trainingcited by 10
  4. 2024Real-time Holistic Robot Pose Estimation with Unknown Statescited by 9
  5. 2024PointPatchRL -- Masked Reconstruction Improves Reinforcement Learning on Point Cloudscited by 5
  6. 2025ROSA: Harnessing Robot States for Vision-Language and Action Alignmentcited by 5
  7. 2024MoVEInt: Mixture of Variational Experts for Learning Human-Robot Interactions from Demonstrationscited by 3
  8. 2025Autonomous Human-Robot Interaction via Operator Imitationcited by 3
  9. 2024Show and Grasp: Few-shot Semantic Segmentation for Robot Grasping through Zero-shot Foundation Modelscited by 2
  10. 2025PointDiffuse: A Dual-Conditional Diffusion Model for Enhanced Point Cloud Semantic Segmentationcited by 2
  11. 2024Depth Restoration of Hand-Held Transparent Objects for Human-to-Robot Handovercited by 1
  12. 2025Learning human-to-robot handovers through 3D scene reconstructioncited by 1
  13. 2025PointNet4D: A Lightweight 4D Point Cloud Video Backbone for Online and Offline Perception in Robotic Applicationscited by 1
  14. 2026HyReach: Vision-Guided Hybrid Manipulator Reaching in Unseen Cluttered Environmentscited by 1
  15. 2025RW-Net: Enhancing Few-Shot Point Cloud Classification with a Wavelet Transform Projection-based Networkcited by 0
  16. 2025Unlocking Hidden Potential in Point Cloud Networks with Attention-Guided Grouping-Feature Coordinationcited by 0
  17. 2025PoseDiff: A Unified Diffusion Model Bridging Robot Pose Estimation and Video-to-Action Controlcited by 0
  18. 2025RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graphcited by 0
  19. 2025Structured Imitation Learning of Interactive Policies through Inverse Gamescited by 0
  20. 2026PhysMani: Physics-principled 3D World Model for Dynamic Object Manipulationcited by 0
  21. 2026SILO: Simulation-in-the-Loop Sim-to-Real Transfer for Multi-Stage Cable Routingcited by 0
  22. 2026Point Ladder Tuning: Parameter-Efficient Hierarchical Adaptation for 3D Point Cloud Understandingcited by 0