# Multi-Task Multi-Frame Visual Piano Transcription

arXiv: https://arxiv.org/abs/2608.03419 · HTML: https://mlsanity.com/paper/2608.03419/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times](https://mlsanity.com/paper/2010.01815/) (2020)
- [Sequence-to-Sequence Piano Transcription with Transformers](https://mlsanity.com/paper/2107.09142/) (2021)
- [Audeo: Audio Generation for a Silent Performance Video](https://mlsanity.com/paper/2006.14348/) (2020)
- [Scoring Time Intervals using Non-Hierarchical Transformer For Automatic Piano Transcription](https://mlsanity.com/paper/2404.09466/) (2024)
- [PianoVAM: A Multimodal Piano Performance Dataset](https://mlsanity.com/paper/2509.08800/) (2025)
- [Quantifying the Corpus Bias Problem in Automatic Music Transcription Systems](https://mlsanity.com/paper/2408.04737/) (2024)
- [Pay Attention to the Keys: Visual Piano Transcription Using Transformers](https://mlsanity.com/paper/2411.09037/) (2024)

## Abstract (arXiv)

Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.