# HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handovers

arXiv: https://arxiv.org/abs/2205.09747 · HTML: https://mlsanity.com/paper/2205.09747/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 54 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 30 of 40)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022)
- [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [Building Generalizable Agents with a Realistic and Rich 3D Environment](https://mlsanity.com/paper/1801.02209/) (2018)
- [ManipulaTHOR: A Framework for Visual Object Manipulation](https://mlsanity.com/paper/2104.11213/) (2021)
- [Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments](https://mlsanity.com/paper/1910.14442/) (2019)
- [IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks](https://mlsanity.com/paper/1911.07246/) (2019)
- [MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments](https://mlsanity.com/paper/1712.03931/) (2017)
- [Assistive Gym: A Physics Simulation Framework for Assistive Robotics](https://mlsanity.com/paper/1910.04700/) (2019)
- [CHALET: Cornell House Agent Learning Environment](https://mlsanity.com/paper/1801.07357/) (2018)
- [A Billion Ways to Grasp: An Evaluation of Grasp Sampling Schemes on a Dense, Physics-based Grasp Data Set](https://mlsanity.com/paper/1912.05604/) (2019)

## Built on by (most cited first, 15 of 15)

- [DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects](https://mlsanity.com/paper/2305.05706/) (2023)
- [Learning Human-to-Robot Handovers from Point Clouds](https://mlsanity.com/paper/2303.17592/) (2023)
- [GraspGF: Learning Score-based Grasping Primitive for Human-assisting Dexterous Grasping](https://mlsanity.com/paper/2309.06038/) (2023)
- [GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation, Demonstration, and Imitation](https://mlsanity.com/paper/2401.00929/) (2024)
- [SynH2R: Synthesizing Hand-Object Motions for Learning Human-to-Robot Handovers](https://mlsanity.com/paper/2311.05599/) (2023)
- [HORT: Monocular Hand-held Objects Reconstruction with Transformers](https://mlsanity.com/paper/2503.21313/) (2025)
- [RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrations](https://mlsanity.com/paper/2502.13134/) (2025)
- [Learning human-to-robot handovers through 3D scene reconstruction](https://mlsanity.com/paper/2507.08726/) (2025)
- [Task-Oriented Human Grasp Synthesis via Context- and Task-Aware Diffusers](https://mlsanity.com/paper/2507.11287/) (2025)
- [PointNet4D: A Lightweight 4D Point Cloud Video Backbone for Online and Offline Perception in Robotic Applications](https://mlsanity.com/paper/2512.01383/) (2025)
- [GenHOI: Generalized Hand-Object Pose Estimation with Occlusion Awareness](https://mlsanity.com/paper/2603.19013/) (2026)
- [Latent Emission-Augmented Perspective-Taking (LEAPT) for Human-Robot Interaction](https://mlsanity.com/paper/2308.06498/) (2023)
- [HRIBench: Benchmarking Interaction-Centric Human-Robot Collaboration](https://mlsanity.com/paper/2607.13056/) (2026)
- [RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction](https://mlsanity.com/paper/2608.13028/) (2026)
- [Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence](https://mlsanity.com/paper/2609.29669/) (2026)

## Abstract (arXiv)

We introduce a new simulation benchmark "HandoverSim" for human-to-robot object handovers. To simulate the giver's motion, we leverage a recent motion capture dataset of hand grasping of objects. We create training and evaluation environments for the receiver with standardized protocols and metrics. We analyze the performance of a set of baselines and show a correlation with a real-world evaluation. Code is open sourced at https://handover-sim.github.io.