# F"urElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance arXiv: https://arxiv.org/abs/2410.05791 ยท HTML: https://mlsanity.com/paper/2410.05791/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 53 references that are in the map. - First posted on arXiv 8 Oct 2024. ## Builds on (most cited first, 28 of 28) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022) - [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021) - [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018) - [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021) - [GRAB: A Dataset of Whole-Body Human Grasping of Objects](https://mlsanity.com/paper/2008.11200/) (2020) - [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022) - [DexYCB: A Benchmark for Capturing Hand Grasping of Objects](https://mlsanity.com/paper/2104.04631/) (2021) - [Reconstructing Hands in 3D with Transformers](https://mlsanity.com/paper/2312.05251/) (2023) - [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022) - [InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/2008.09309/) (2020) - [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017) - [Object Motion Guided Human Motion Synthesis](https://mlsanity.com/paper/2309.16237/) (2023) - [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022) - [Embodied Hands: Modeling and Capturing Hands and Bodies Together](https://mlsanity.com/paper/2201.02610/) (2022) - [DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation](https://mlsanity.com/paper/2403.07788/) (2024) - [Learning human behaviors from motion capture by adversarial imitation](https://mlsanity.com/paper/1707.02201/) (2017) - [High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times](https://mlsanity.com/paper/2010.01815/) (2020) - [Audio to Body Dynamics](https://mlsanity.com/paper/1712.09382/) (2017) - [Hierarchical Planning and Control for Box Loco-Manipulation](https://mlsanity.com/paper/2306.09532/) (2023) - [GeneOH Diffusion: Towards Generalizable Hand-Object Interaction Denoising via Denoising Diffusion](https://mlsanity.com/paper/2402.14810/) (2024) - [RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.04150/) (2023) - [Learning to Use Chopsticks in Diverse Gripping Styles](https://mlsanity.com/paper/2205.14313/) (2022) - [Composite Motion Learning with Task Control](https://mlsanity.com/paper/2305.03286/) (2023) - [Temporally Guided Music-to-Body-Movement Generation](https://mlsanity.com/paper/2009.08015/) (2020) - [Towards Learning to Play Piano with Dexterous Hands and Touch](https://mlsanity.com/paper/2106.02040/) (2021) - [A GAN-Like Approach for Physics-Based Imitation Learning and Interactive Character Control](https://mlsanity.com/paper/2105.10066/) (2021) - [Synthesize Dexterous Nonprehensile Pregrasp for Ungraspable Objects](https://mlsanity.com/paper/2305.04654/) (2023) ## Built on by (most cited first, 10 of 10) - [InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactions](https://mlsanity.com/paper/2502.20390/) (2025) - [Predicting 4D Hand Trajectory from Monocular Videos](https://mlsanity.com/paper/2501.08329/) (2025) - [Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesis](https://mlsanity.com/paper/2504.09885/) (2025) - [Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty](https://mlsanity.com/paper/2602.18312/) (2026) - [HandX: Scaling Bimanual Motion and Interaction Generation](https://mlsanity.com/paper/2603.28766/) (2026) - [MUSIC: Learning Muscle-Driven Dexterous Hand Control](https://mlsanity.com/paper/2604.23886/) (2026) - [Robo-Saber: Generating and Simulating Virtual Reality Players](https://mlsanity.com/paper/2602.18319/) (2026) - [HandelBot: Real-World Piano Playing via Fast Adaptation of Dexterous Robot Policies](https://mlsanity.com/paper/2603.12243/) (2026) - [PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination](https://mlsanity.com/paper/2604.12856/) (2026) - [Generalized Audio-Driven Synthesis of Precise Drummer Motion](https://mlsanity.com/paper/2608.19055/) (2026) ## Abstract (arXiv) Piano playing requires agile, precise, and coordinated hand control that stretches the limits of dexterity. Hand motion models with the sophistication to accurately recreate piano playing have a wide range of applications in character animation, embodied AI, biomechanics, and VR/AR. In this paper, we construct a first-of-its-kind large-scale dataset that contains approximately 10 hours of 3D hand motion and audio from 15 elite-level pianists playing 153 pieces of classical music. To capture natural performances, we designed a markerless setup in which motions are reconstructed from multi-view videos using state-of-the-art pose estimation models. The motion data is further refined via inverse kinematics using the high-resolution MIDI key-pressing data obtained from sensors in a specialized Yamaha Disklavier piano. Leveraging the collected dataset, we developed a pipeline that can synthesize physically-plausible hand motions for musical scores outside of the dataset. Our approach employs a combination of imitation learning and reinforcement learning to obtain policies for physics-based bimanual control involving the interaction between hands and piano keys. To solve the sampling efficiency problem with the large motion dataset, we use a diffusion model to generate natural reference motions, which provide high-level trajectory and fingering (finger order and placement) information. However, the generated reference motion alone does not provide sufficient accuracy for piano performance modeling. We then further augmented the data by using musical similarity to retrieve similar motions from the captured dataset to boost the precision of the RL policy. With the proposed method, our model generates natural, dexterous motions that generalize to music from outside the training dataset.