# Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion

arXiv: https://arxiv.org/abs/2006.15561 · HTML: https://mlsanity.com/paper/2006.15561/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2020.

## Built on by (most cited first, 30 of 39)

- [MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video](https://mlsanity.com/paper/2203.00859/) (2022)
- [Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time](https://mlsanity.com/paper/2106.05266/) (2021)
- [What's in your hands? 3D Reconstruction of Generic Objects in Hands](https://mlsanity.com/paper/2204.07153/) (2022)
- [TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinement](https://mlsanity.com/paper/2205.07982/) (2022)
- [Model-based 3D Hand Reconstruction via Self-Supervised Learning](https://mlsanity.com/paper/2103.11703/) (2021)
- [I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modeling](https://mlsanity.com/paper/2102.03725/) (2021)
- [SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation](https://mlsanity.com/paper/2308.11509/) (2023)
- [EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2411.14280/) (2024)
- [Consistent 3D Hand Reconstruction in Video via self-supervised Learning](https://mlsanity.com/paper/2201.09548/) (2022)
- [Hand-Object Interaction Image Generation](https://mlsanity.com/paper/2211.15663/) (2022)
- [Tracking and Reconstructing Hand Object Interactions from Point Cloud Sequences in the Wild](https://mlsanity.com/paper/2209.12009/) (2022)
- [DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field](https://mlsanity.com/paper/2308.08231/) (2023)
- [ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimation](https://mlsanity.com/paper/2305.01618/) (2023)
- [Dense Hand-Object(HO) GraspNet with Full Grasping Taxonomy and Dynamics](https://mlsanity.com/paper/2409.04033/) (2024)
- [HOISDF: Constraining 3D Hand-Object Pose Estimation with Global Signed Distance Fields](https://mlsanity.com/paper/2402.17062/) (2024)
- [Sequential Voting with Relational Box Fields for Active Object Detection](https://mlsanity.com/paper/2110.11524/) (2021)
- [MOHO: Learning Single-view Hand-held Object Reconstruction with Multi-view Occlusion-Aware Supervision](https://mlsanity.com/paper/2310.11696/) (2023)
- [CHORD: Category-level Hand-held Object Reconstruction via Shape Deformation](https://mlsanity.com/paper/2308.10574/) (2023)
- [Deep learning for 3D human pose estimation and mesh recovery: A survey](https://mlsanity.com/paper/2402.18844/) (2024)
- [PEAR: Phrase-Based Hand-Object Interaction Anticipation](https://mlsanity.com/paper/2407.21510/) (2024)
- [BIGS: Bimanual Category-agnostic Interaction Reconstruction from Monocular Videos via 3D Gaussian Splatting](https://mlsanity.com/paper/2504.09097/) (2025)
- [DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image](https://mlsanity.com/paper/2406.17988/) (2024)
- [Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidance](https://mlsanity.com/paper/2508.18213/) (2025)
- [Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling](https://mlsanity.com/paper/2401.15987/) (2024)
- [PHRIT: Parametric Hand Representation with Implicit Template](https://mlsanity.com/paper/2309.14916/) (2023)
- [Sparse multi-view hand-object reconstruction for unseen environments](https://mlsanity.com/paper/2405.01353/) (2024)
- [Latent Adaptive Planner for Dynamic Manipulation](https://mlsanity.com/paper/2505.03077/) (2025)
- [ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions](https://mlsanity.com/paper/2603.25791/) (2026)
- [D-SCo: Dual-Stream Conditional Diffusion for Monocular Hand-Held Object Reconstruction](https://mlsanity.com/paper/2311.14189/) (2023)
- [Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery](https://mlsanity.com/paper/2505.17677/) (2025)

## Abstract (arXiv)

Accurate 3D reconstruction of the hand and object shape from a hand-object image is important for understanding human-object interaction as well as human daily activities. Different from bare hand pose estimation, hand-object interaction poses a strong constraint on both the hand and its manipulated object, which suggests that hand configuration may be crucial contextual information for the object, and vice versa. However, current approaches address this task by training a two-branch network to reconstruct the hand and object separately with little communication between the two branches. In this work, we propose to consider hand and object jointly in feature space and explore the reciprocity of the two branches. We extensively investigate cross-branch feature fusion architectures with MLP or LSTM units. Among the investigated architectures, a variant with LSTM units that enhances object feature with hand feature shows the best performance gain. Moreover, we employ an auxiliary depth estimation module to augment the input RGB image with the estimated depth map, which further improves the reconstruction accuracy. Experiments conducted on public datasets demonstrate that our approach significantly outperforms existing approaches in terms of the reconstruction accuracy of objects.