mlsanityData to 7 Oct 2026

Towards Learning to Play Piano with Dexterous Hands and Touch

arXiv 2106.02040 ↗First posted 3 Jun 2021cs.RO cs.AI stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

The virtuoso plays the piano with passion, poetry and extraordinary technical ability. As Liszt said (a virtuoso)must call up scent and blossom, and breathe the breath of life. The strongest robots that can play a piano are based on a combination of specialized robot hands/piano and hardcoded planning algorithms. In contrast to that, in this paper, we demonstrate how an agent can learn directly from machine-readable music score to play the piano with dexterous hands on a simulated piano using reinforcement learning (RL) from scratch. We demonstrate the RL agents can not only find the correct key position but also deal with various rhythmic, volume and fingering, requirements. We achieve this by using a touch-augmented reward and a novel curriculum of tasks. We conclude by carefully studying the important aspects to enable such learning algorithms and that can potentially shed light on future research in this direction.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 13 of its 36 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

13 of 36 references.

References · 13

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2018Learning Dexterous In-Hand Manipulationcited by 643
  9. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 164
  10. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 100
  11. 2020TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensorscited by 59
  12. 2019Cable Manipulation with a Tactile-Reactive Grippercited by 44
1 more
  1. 2021SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulationcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 6; 2025: 4; 2026: 1.

14 papers.

Sort:

By area:

  1. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  2. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  3. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  4. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  5. 2023RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learningcited by 29
  6. 2024Lessons from Learning to Spin "Pens"cited by 22
  7. 2024PianoMime: Learning a Generalist, Dexterous Piano Player from Internet Demonstrationscited by 8
  8. 2024F"urElise: Capturing and Physically Synthesizing Hand Motions of Piano Performancecited by 8
  9. 2025Separate to Collaborate: Dual-Stream Diffusion Model for Coordinated Piano Hand Motion Synthesiscited by 4
  10. 2024RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Handscited by 3
  11. 2025Learning Gentle Grasping Using Vision, Sound, and Touchcited by 2
  12. 2025PANDORA: Diffusion Policy Learning for Dexterous Robotic Piano Playingcited by 2
2 more
  1. 2025Learning to Play Piano in the Real Worldcited by 1
  2. 2026MUSIC: Learning Muscle-Driven Dexterous Hand Controlcited by 1