# Towards Efficient and Real-Time Piano Transcription Using Neural Autoregressive Models

arXiv: https://arxiv.org/abs/2404.06818 · HTML: https://mlsanity.com/paper/2404.06818/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 41 references that are in the map.
- First posted on arXiv 10 Apr 2024.

## Builds on (most cited first, 14 of 14)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times](https://mlsanity.com/paper/2010.01815/) (2020)
- [Sequence-to-Sequence Piano Transcription with Transformers](https://mlsanity.com/paper/2107.09142/) (2021)
- [On the Potential of Simple Framewise Approaches to Piano Transcription](https://mlsanity.com/paper/1612.05153/) (2016)
- [ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data](https://mlsanity.com/paper/2107.04954/) (2021)
- [Deep Polyphonic ADSR Piano Note Transcription](https://mlsanity.com/paper/1906.09165/) (2019)
- [Adversarial Learning for Improved Onsets and Frames Music Transcription](https://mlsanity.com/paper/1906.08512/) (2019)
- [Polyphonic Piano Transcription Using Autoregressive Multi-State Note Model](https://mlsanity.com/paper/2010.01104/) (2020)
- [Deep Unsupervised Drum Transcription](https://mlsanity.com/paper/1906.03697/) (2019)

## Built on by (most cited first, 4 of 4)

- [D3RM: A Discrete Denoising Diffusion Refinement Model for Piano Transcription](https://mlsanity.com/paper/2501.05068/) (2025)
- [Towards Robust Transcription: Exploring Noise Injection Strategies for Training Data Augmentation](https://mlsanity.com/paper/2410.14122/) (2024)
- [Dialogue in Resonance: An Interactive Music Piece for Piano and Real-Time Automatic Transcription System](https://mlsanity.com/paper/2505.16259/) (2025)
- [TCPα: Margin-Controlled Confidence estimation for reliable Music Information Retrieval](https://mlsanity.com/paper/2608.20326/) (2026)

## Abstract (arXiv)

In recent years, advancements in neural network designs and the availability of large-scale labeled datasets have led to significant improvements in the accuracy of piano transcription models. However, most previous work focused on high-performance offline transcription, neglecting deliberate consideration of model size. The goal of this work is to implement real-time inference for piano transcription while ensuring both high performance and lightweight. To this end, we propose novel architectures for convolutional recurrent neural networks, redesigning an existing autoregressive piano transcription model. First, we extend the acoustic module by adding a frequency-conditioned FiLM layer to the CNN module to adapt the convolutional filters on the frequency axis. Second, we improve note-state sequence modeling by using a pitchwise LSTM that focuses on note-state transitions within a note. In addition, we augment the autoregressive connection with an enhanced recursive context. Using these components, we propose two types of models; one for high performance and the other for high compactness. Through extensive experiments, we show that the proposed models are comparable to state-of-the-art models in terms of note accuracy on the MAESTRO dataset. We also investigate the effective model size and real-time inference latency by gradually streamlining the architecture. Finally, we conduct cross-data evaluation on unseen piano datasets and in-depth analysis to elucidate the effect of the proposed components in the view of note length and pitch range.