mlsanityData to 7 Oct 2026

RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands

arXiv 2408.11048 ↗First posted 20 Aug 2024cs.RO cs.AI cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

It has been a long-standing research goal to endow robot hands with human-level dexterity. Bi-manual robot piano playing constitutes a task that combines challenges from dynamic tasks, such as generating fast while precise motions, with slower but contact-rich manipulation problems. Although reinforcement learning based approaches have shown promising results in single-task performance, these methods struggle in a multi-song setting. Our work aims to close this gap and, thereby, enable imitation learning approaches for robot piano playing at scale. To this end, we introduce the Robot Piano 1 Million (RP1M) dataset, containing bi-manual robot piano playing motion data of more than one million trajectories. We formulate finger placements as an optimal transport problem, thus, enabling automatic annotation of vast amounts of unlabeled songs. Benchmarking existing imitation learning approaches shows that such approaches reach state-of-the-art robot piano playing performance by leveraging RP1M.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 59 of its 90 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

59 of 90 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  2. 2022Behavior Transformers: Cloning k modes with one stone22 Jun
  3. 2020State-Only Imitation Learning for Dexterous Manipulation7 Apr

Other references · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  9. 2022Video Diffusion Modelscited by 2,280
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  12. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
38 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  2. 2018DeepMind Control Suitecited by 935
  3. 2022A Generalist Agentcited by 802
  4. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  5. 2019Solving Rubik's Cube with a Robot Handcited by 699
  6. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2018Learning Dexterous In-Hand Manipulationcited by 643
  9. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  10. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  11. 2021Implicit Behavioral Cloningcited by 442
  12. 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 306
  13. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  14. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  15. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  16. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  17. 2021A System for General In-Hand Object Re-Orientationcited by 155
  18. 2022DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulationcited by 129
  19. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 117
  20. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  21. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  22. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  23. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  24. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 71
  25. 2024DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasetscited by 47
  26. 2020Deep Differentiable Grasp Planner for High-DOF Gripperscited by 39
  27. 2020GiantMIDI-Piano: A large-scale MIDI dataset for classical piano musiccited by 37
  28. 2024Twisting Lids Off with Two Handscited by 36
  29. 2021Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFingercited by 34
  30. 2024RealDex: Towards Human-like Grasping for Robotic Dexterous Handcited by 33
  31. 2016Learning Dexterous Manipulation Policies from Experience and Imitationcited by 32
  32. 2020Learning to Play Table Tennis From Scratch using Muscular Robotscited by 31
  33. 2021RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learningcited by 31
  34. 2022Learning to Use Chopsticks in Diverse Gripping Stylescited by 29
  35. 2023RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learningcited by 29
  36. 2022PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Trainingcited by 24
  37. 2020Learning Dexterous Manipulation from Suboptimal Expertscited by 23
  38. 2024MultiGripperGrasp: A Dataset for Robotic Grasping from Parallel Jaw Grippers to Dexterous Handscited by 16

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Dex1B: Learning with 1B Demonstrations for Dexterous Manipulationcited by 23
  2. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Datacited by 1
  3. 2025Learning to Play Piano in the Real Worldcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Dexterous Robotic Piano Playing at Scale ↗
  2. 2024F"urElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance
  3. 2025PANDORA: Diffusion Policy Learning for Dexterous Robotic Piano Playing
  4. 2025Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesis
  5. 2026PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination
  6. 2025RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments
  7. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
  8. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning