Learning human-to-robot handovers through 3D scene reconstruction
Not on the map. Why?
Abstract (arXiv)
Learning robot manipulation policies from raw, real-world image data requires a large number of robot-action trials in the physical environment. Although training using simulations offers a cost-effective alternative, the visual domain gap between simulation and robot workspace remains a major limitation. Gaussian Splatting visual reconstruction methods have recently provided new directions for robot manipulation by generating realistic environments. In this paper, we propose the first method for learning supervised-based robot handovers solely from RGB images without the need of real-robot training or real-robot data collection. The proposed policy learner, Human-to-Robot Handover using Sparse-View Gaussian Splatting (H2RH-SGS), leverages sparse-view Gaussian Splatting reconstruction of human-to-robot handover scenes to generate robot demonstrations containing image-action pairs captured with a camera mounted on the robot gripper. As a result, the simulated camera pose changes in the reconstructed scene can be directly translated into gripper pose changes. We train a robot policy on demonstrations collected with 16 household objects and em directly deploy this policy in the real environment. Experiments in both Gaussian Splatting reconstructed scene and real-world human-to-robot handover experiments demonstrate that H2RH-SGS serves as a new and effective representation for the human-to-robot handover task.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 11 Jul 2025.
Builds on
44 of 51 references.
References · 44
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
- 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,082
- 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
- 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
- 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
- 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
- 2023Fast Segment Anythingcited by 290
- 2020Understanding Human Hands in Contact at Internet Scalecited by 281
32 more
- 2023FSGS: Real-Time Few-shot View Synthesis using Gaussian Splattingcited by 279
- 2024DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalizationcited by 256
- 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 242
- 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
- 2018Progressive Feature Alignment for Unsupervised Domain Adaptationcited by 183
- 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 174
- 2021FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integrationcited by 169
- 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
- 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 108
- 2015Learning to See by Movingcited by 102
- 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 94
- 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 91
- 2024CoherentGS: Sparse Novel View Synthesis with Coherent 3D Gaussianscited by 62
- 2024Splat-Nav: Safe Real-Time Robot Navigation in Gaussian Splatting Mapscited by 55
- 2020Deep Imitation Learning for Bimanual Robotic Manipulationcited by 50
- 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49
- 2022NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fieldscited by 44
- 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
- 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
- 2024Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splattingcited by 38
- 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 38
- 2023Learning Human-to-Robot Handovers from Point Cloudscited by 34
- 2024ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedbackcited by 31
- 2024MVPGS: Excavating Multi-view Priors for Gaussian Splatting from Sparse Input Viewscited by 29
- 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Controlcited by 24
- 2023Learning to Read Braille: Bridging the Tactile Reality Gap with Diffusion Modelscited by 18
- 2023End-to-end Reinforcement Learning for Time-Optimal Quadcopter Flightcited by 17
- 2022HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handoverscited by 15
- 2024GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation, Demonstration, and Imitationcited by 14
- 2024VTON-HandFit: Virtual Try-on for Arbitrary Hand Pose Guided by Hand Priors Embeddingcited by 11
- 2023Sim2Real Bilevel Adaptation for Object Surface Classification using Vision-Based Tactile Sensorscited by 4
- 2021OHPL: One-shot Hand-eye Policy Learnercited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.