# PianoVAM: A Multimodal Piano Performance Dataset

arXiv: https://arxiv.org/abs/2509.08800 · HTML: https://mlsanity.com/paper/2509.08800/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Builds on (most cited first, 12 of 12)

- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [MediaPipe Hands: On-device Real-time Hand Tracking](https://mlsanity.com/paper/2006.10214/) (2020)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [ViTPose++: Vision Transformer for Generic Body Pose Estimation](https://mlsanity.com/paper/2212.04246/) (2022)
- [High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times](https://mlsanity.com/paper/2010.01815/) (2020)
- [Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mamba](https://mlsanity.com/paper/2407.09646/) (2024)
- [A Data-Driven Analysis of Robust Automatic Piano Transcription](https://mlsanity.com/paper/2402.01424/) (2024)
- [Scoring Time Intervals using Non-Hierarchical Transformer For Automatic Piano Transcription](https://mlsanity.com/paper/2404.09466/) (2024)
- [A cappella: Audio-visual Singing Voice Separation](https://mlsanity.com/paper/2104.09946/) (2021)
- [Statistical Learning and Estimation of Piano Fingering](https://mlsanity.com/paper/1904.10237/) (2019)
- [HPPNet: Modeling the Harmonic Structure and Pitch Invariance in Piano Transcription](https://mlsanity.com/paper/2208.14339/) (2022)
- [Towards Robust Transcription: Exploring Noise Injection Strategies for Training Data Augmentation](https://mlsanity.com/paper/2410.14122/) (2024)

## Built on by (most cited first, 5 of 5)

- [Two Web Toolkits for Multimodal Piano Performance Dataset Acquisition and Fingering Annotation](https://mlsanity.com/paper/2509.15222/) (2025)
- [Tipiano: Cascaded Piano Hand Motion Synthesis via Fingertip Priors](https://mlsanity.com/paper/2604.09692/) (2026)
- [Profy: Interpretable Visualization of Expertise-Dependent Motor Skills Toward Supporting Piano Practice](https://mlsanity.com/paper/2606.10627/) (2026)
- [Multi-Task Multi-Frame Visual Piano Transcription](https://mlsanity.com/paper/2608.03419/) (2026)
- [Synthesis and editing of multi-instrument audio mixtures using scalar-quantised latents with MIDI Span conditioning](https://mlsanity.com/paper/2609.25546/) (2026)

## Abstract (arXiv)

The multimodal nature of music performance has driven increasing interest in data beyond the audio domain within the music information retrieval (MIR) community. This paper introduces PianoVAM, a comprehensive piano performance dataset that includes videos, audio, MIDI, hand landmarks, fingering labels, and rich metadata. The dataset was recorded using a Disklavier piano, capturing audio and MIDI from amateur pianists during their daily practice sessions, alongside synchronized top-view videos in realistic and varied performance conditions. Hand landmarks and fingering labels were extracted using a pretrained hand pose estimation model and a semi-automated fingering annotation algorithm. We discuss the challenges encountered during data collection and the alignment process across different modalities. Additionally, we describe our fingering annotation method based on hand landmarks extracted from videos. Finally, we present benchmarking results for both audio-only and audio-visual piano transcription using the PianoVAM dataset and discuss additional potential applications.