# RenderIH: A Large-scale Synthetic Dataset for 3D Interacting Hand Pose Estimation

arXiv: https://arxiv.org/abs/2309.09301 · HTML: https://mlsanity.com/paper/2309.09301/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 17 Sep 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021)
- [ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation](https://mlsanity.com/paper/2204.13662/) (2022)
- [Resolving 3D Human Pose Ambiguities with 3D Scene Constraints](https://mlsanity.com/paper/1908.06963/) (2019)
- [FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images](https://mlsanity.com/paper/1909.04349/) (2019)
- [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020)
- [Coherent Reconstruction of Multiple Humans from a Single Image](https://mlsanity.com/paper/2006.08586/) (2020)
- [CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction](https://mlsanity.com/paper/2012.00924/) (2020)
- [Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation](https://mlsanity.com/paper/2104.14639/) (2021)
- [Capturing Hands in Action using Discriminative Salient Points and Physics Simulation](https://mlsanity.com/paper/1506.02178/) (2015)
- [Interacting Attention Graph for Single Image Two-Hand Reconstruction](https://mlsanity.com/paper/2203.09364/) (2022)
- [Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor](https://mlsanity.com/paper/1704.02201/) (2017)
- [Encoder-decoder with Multi-level Attention for 3D Human Shape and Pose Estimation](https://mlsanity.com/paper/2109.02303/) (2021)
- [Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction](https://mlsanity.com/paper/1906.04158/) (2019)
- [Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation](https://mlsanity.com/paper/2107.00434/) (2021)
- [3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal](https://mlsanity.com/paper/2207.11061/) (2022)
- [3D Hand Pose Tracking and Estimation Using Stereo Matching](https://mlsanity.com/paper/1610.07214/) (2016)
- [Two-hand Global 3D Pose Estimation Using Monocular RGB](https://mlsanity.com/paper/2006.01320/) (2020)
- [DART: Articulated Hand Model with Diverse Accessories and Rich Textures](https://mlsanity.com/paper/2210.07650/) (2022)
- [H2O: A Benchmark for Visual Human-human Object Handover Analysis](https://mlsanity.com/paper/2104.11466/) (2021)
- [MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation](https://mlsanity.com/paper/2012.03206/) (2020)

## Built on by (most cited first, 16 of 16)

- [HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video](https://mlsanity.com/paper/2311.18448/) (2023)
- [Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects](https://mlsanity.com/paper/2403.16428/) (2024)
- [MagicHOI: Leveraging 3D Priors for Accurate Hand-object Reconstruction from Short Monocular Video Clips](https://mlsanity.com/paper/2508.05506/) (2025)
- [HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models](https://mlsanity.com/paper/2312.04867/) (2023)
- [AttentionHand: Text-driven Controllable Hand Image Generation for 3D Hand Reconstruction in the Wild](https://mlsanity.com/paper/2407.18034/) (2024)
- [FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation](https://mlsanity.com/paper/2412.02690/) (2024)
- [BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics](https://mlsanity.com/paper/2312.07937/) (2023)
- [Deep Learning Approaches for Seizure Video Analysis: A Review](https://mlsanity.com/paper/2312.10930/) (2023)
- [Weakly-Supervised 3D Hand Reconstruction with Knowledge Prior and Uncertainty Guidance](https://mlsanity.com/paper/2407.12307/) (2024)
- [Text-Driven 3D Hand Motion Generation from Sign Language Data](https://mlsanity.com/paper/2508.15902/) (2025)
- [SHOW3D: Capturing Scenes of 3D Hands and Objects in the Wild](https://mlsanity.com/paper/2603.28760/) (2026)
- [AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation](https://mlsanity.com/paper/2603.25726/) (2026)
- [Visuo-Acoustic Hand Pose and Contact Estimation](https://mlsanity.com/paper/2508.00852/) (2025)
- [DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation](https://mlsanity.com/paper/2510.15786/) (2025)
- [SesaHand: Enhancing 3D Hand Reconstruction via Controllable Generation with Semantic and Structural Alignment](https://mlsanity.com/paper/2603.00443/) (2026)
- [Prompt-to-Gesture: Measuring the Capabilities of Image-to-Video Deictic Gesture Generation](https://mlsanity.com/paper/2604.14953/) (2026)

## Abstract (arXiv)

The current interacting hand (IH) datasets are relatively simplistic in terms of background and texture, with hand joints being annotated by a machine annotator, which may result in inaccuracies, and the diversity of pose distribution is limited. However, the variability of background, pose distribution, and texture can greatly influence the generalization ability. Therefore, we present a large-scale synthetic dataset RenderIH for interacting hands with accurate and diverse pose annotations. The dataset contains 1M photo-realistic images with varied backgrounds, perspectives, and hand textures. To generate natural and diverse interacting poses, we propose a new pose optimization algorithm. Additionally, for better pose estimation accuracy, we introduce a transformer-based pose estimation network, TransHand, to leverage the correlation between interacting hands and verify the effectiveness of RenderIH in improving results. Our dataset is model-agnostic and can improve more accuracy of any hand pose estimation method in comparison to other real or synthetic datasets. Experiments have shown that pretraining on our synthetic data can significantly decrease the error from 6.76mm to 5.79mm, and our Transhand surpasses contemporary methods. Our dataset and code are available at https://github.com/adwardlee/RenderIH.