# Profy: Interpretable Visualization of Expertise-Dependent Motor Skills Toward Supporting Piano Practice

arXiv: https://arxiv.org/abs/2606.10627 · HTML: https://mlsanity.com/paper/2606.10627/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 70 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Adaptive pooling operators for weakly labeled sound event detection](https://mlsanity.com/paper/1804.10070/) (2018)
- [Piano Genie](https://mlsanity.com/paper/1810.05246/) (2018)
- [An Interdisciplinary Review of Music Performance Analysis](https://mlsanity.com/paper/2104.09018/) (2021)
- [PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance](https://mlsanity.com/paper/2406.09326/) (2024)
- [LLaQo: Towards a Query-Based Coach in Expressive Music Performance Assessment](https://mlsanity.com/paper/2409.08795/) (2024)
- [PianoVAM: A Multimodal Piano Performance Dataset](https://mlsanity.com/paper/2509.08800/) (2025)
- [Score-informed Networks for Music Performance Assessment](https://mlsanity.com/paper/2008.00203/) (2020)
- [Real-Time Audio-to-Score Alignment of Music Performances Containing Errors and Arbitrary Repeats and Skips](https://mlsanity.com/paper/1512.07748/) (2015)
- [Getting Closer to the Essence of Music: The Con Espressione Manifesto](https://mlsanity.com/paper/1611.09733/) (2016)
- [From Audio Encoders to Piano Judges: Benchmarking Performance Understanding for Solo Piano](https://mlsanity.com/paper/2407.04518/) (2024)
- [A Survey of Augmented Piano Prototypes: Has Augmentation Improved Learning Experiences?](https://mlsanity.com/paper/2208.09929/) (2022)
- [DDSupport: Language Learning Support System that Displays Differences and Distances from Model Speech](https://mlsanity.com/paper/2212.04930/) (2022)

## Abstract (arXiv)

The quality of piano performance depends on nuanced timing, articulation, and dynamic control, but practice feedback is often summary-based and hard to act on. We introduce Profy, a weakly supervised system that learns from take-level labels derived from aggregated listener ratings (expert-labeled vs. amateur-labeled) to produce time-aligned highlights for review during piano practice. We collected synchronized 1 kHz key-motion and audio from 73 pianists and used 1,083 valid takes for modeling and evaluation. The model outputs clip-level predictions together with evidence scores on a shared resampled model time base for visualization. On 20 amateur clips from short technique studies annotated by 21 expert pianists, the displayed highlight score aligns with passages that expert pianists marked for review despite training without localized labels (Pearson r=0.61, ROC-AUC 0.75). Rather than summarizing a take with a single global score, Profy helps learners decide where to inspect next by supporting scrubbing, looping, and focused replay of time-localized passages associated with expert-amateur differences.