HandX: Scaling Bimanual Motion and Interaction Generation
Not on the map. Why?
Abstract (arXiv)
Synthesizing human motion has advanced rapidly, yet realistic hand motion and bimanual interaction remain underexplored. Whole-body models often miss the fine-grained cues that drive dexterous behavior, finger articulation, contact timing, and inter-hand coordination, and existing resources lack high-fidelity bimanual sequences that capture nuanced finger dynamics and collaboration. To fill this gap, we present HandX, a unified foundation spanning data, annotation, and evaluation. We consolidate and filter existing datasets for quality, and collect a new motion-capture dataset targeting underrepresented bimanual interactions with detailed finger dynamics. For scalable annotation, we introduce a decoupled strategy that extracts representative motion features, e.g., contact events and finger flexion, and then leverages reasoning from large language models to produce fine-grained, semantically rich descriptions aligned with these features. Building on the resulting data and annotations, we benchmark diffusion and autoregressive models with versatile conditioning modes. Experiments demonstrate high-quality dexterous motion generation, supported by our newly proposed hand-focused metrics. We further observe clear scaling trends: larger models trained on larger, higher-quality datasets produce more semantically coherent bimanual motion. Our dataset is released to support future research.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 70 of its 87 references that are in the map. First posted on arXiv 30 Mar 2026.
Builds on
70 of 87 references.
References · 70
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2022Human Motion Diffusion Modelcited by 1,080
- 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 780
- 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 588
- 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
- 2022TEMOS: Generating diverse human motions from textual descriptionscited by 492
- 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
- 2023Human Motion Diffusion as a Generative Priorcited by 357
- 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
- 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
38 more
- 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 287
- 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
- 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
- 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
- 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
- 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
- 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 198
- 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 182
- 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 166
- 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 129
- 2024HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videoscited by 129
- 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 111
- 2017Text2Action: Generative Adversarial Synthesis from Language to Actioncited by 106
- 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
- 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 104
- 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 88
- 2024Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interactioncited by 86
- 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 86
- 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
- 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 78
- 2022TOCH: Spatio-Temporal Object-to-Hand Correspondence for Motion Refinementcited by 70
- 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 68
- 2025HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videoscited by 66
- 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 64
- 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 62
- 2024Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformercited by 61
- 2024ParCo: Part-Coordinating Text-to-Motion Synthesiscited by 55
- 2024ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Modelcited by 53
- 2024Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Cameracited by 51
- 2023Neural Sign Actors: A diffusion model for 3D sign language production from textcited by 49
- 2025GestureLSM: Latent Shortcut based Co-Speech Gesture Generation with Spatial-Temporal Modelingcited by 44
- 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 33
- 2025HOIGPT: Learning Long Sequence Hand-Object Interaction with Language Modelscited by 33
- 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 32
- 2024BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objectscited by 30
- 2023Stochastic Multi-Person 3D Motion Forecastingcited by 28
- 2023GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistencycited by 27
- 2023MotionScript: Natural Language Descriptions for Expressive 3D Human Motionscited by 21
50 most cited of 70 All 70 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024AsymDex: Asymmetry and Relative Coordinates for RL-based Bimanual Dexterity
- 2024ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection
- 2026PatternDex: Learning Interaction Patterns to Guide Reinforcement Learning of Bimanual Dexterous Manipulation of Articulated Objects
- 2026Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation
- 2025TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactions
- 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning
- 2026Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands
- 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videos