# Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics

arXiv: https://arxiv.org/abs/2007.12287 · HTML: https://mlsanity.com/paper/2007.12287/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2020.

## Built on by (most cited first, 30 of 34)

- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration](https://mlsanity.com/paper/2108.06428/) (2021)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023)
- [From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations](https://mlsanity.com/paper/2401.01885/) (2024)
- [Driving-Signal Aware Full-Body Avatars](https://mlsanity.com/paper/2105.10441/) (2021)
- [MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image](https://mlsanity.com/paper/2112.02753/) (2021)
- [TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinement](https://mlsanity.com/paper/2205.07982/) (2022)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation](https://mlsanity.com/paper/2305.18891/) (2023)
- [Affective Faces for Goal-Driven Dyadic Communication](https://mlsanity.com/paper/2301.10939/) (2023)
- [BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization](https://mlsanity.com/paper/2302.05075/) (2023)
- [The One Where They Reconstructed 3D Humans and Environments in TV Shows](https://mlsanity.com/paper/2207.14279/) (2022)
- [HARP: Personalized Hand Reconstruction from a Monocular RGB Video](https://mlsanity.com/paper/2212.09530/) (2022)
- [Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements](https://mlsanity.com/paper/2111.00763/) (2021)
- [Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation](https://mlsanity.com/paper/2207.12817/) (2022)
- [Diverse 3D Hand Gesture Prediction from Body Dynamics by Bilateral Hand Disentanglement](https://mlsanity.com/paper/2303.01765/) (2023)
- [REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning](https://mlsanity.com/paper/2504.04956/) (2025)
- [MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition](https://mlsanity.com/paper/2405.20666/) (2024)
- [HandOS: 3D Hand Reconstruction in One Stage](https://mlsanity.com/paper/2412.01537/) (2024)
- [Spatial-Temporal Parallel Transformer for Arm-Hand Dynamic Estimation](https://mlsanity.com/paper/2203.16202/) (2022)
- [BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics](https://mlsanity.com/paper/2312.07937/) (2023)
- [InterAct: Capture and Modelling of Realistic, Expressive and Interactive Activities between Two Persons in Daily Scenarios](https://mlsanity.com/paper/2405.11690/) (2024)
- [Multi-View Video-Based 3D Hand Pose Estimation](https://mlsanity.com/paper/2109.11747/) (2021)
- [Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling](https://mlsanity.com/paper/2401.15987/) (2024)
- [Text-Driven 3D Hand Motion Generation from Sign Language Data](https://mlsanity.com/paper/2508.15902/) (2025)
- [Learning Continuous Mesh Representation with Spherical Implicit Surface](https://mlsanity.com/paper/2301.04695/) (2023)
- [VM-BHINet:Vision Mamba Bimanual Hand Interaction Network for 3D Interacting Hand Mesh Recovery From a Single RGB Image](https://mlsanity.com/paper/2504.14618/) (2025)
- [InterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarios](https://mlsanity.com/paper/2509.05747/) (2025)
- [Topic Detection in Continuous Sign Language Videos](https://mlsanity.com/paper/2209.02402/) (2022)

## Abstract (arXiv)

We propose a novel learned deep prior of body motion for 3D hand shape synthesis and estimation in the domain of conversational gestures. Our model builds upon the insight that body motion and hand gestures are strongly correlated in non-verbal communication settings. We formulate the learning of this prior as a prediction task of 3D hand shape over time given body motion input alone. Trained with 3D pose estimations obtained from a large-scale dataset of internet videos, our hand prediction model produces convincing 3D hand gestures given only the 3D motion of the speaker's arms as input. We demonstrate the efficacy of our method on hand gesture synthesis from body motion input, and as a strong body prior for single-view image-based 3D hand pose estimation. We demonstrate that our method outperforms previous state-of-the-art approaches and can generalize beyond the monologue-based training data to multi-person conversations. Video results are available at http://people.eecs.berkeley.edu/~evonne_ng/projects/body2hands/.