# Automatic Piano Transcription with Hierarchical Frequency-Time Transformer

arXiv: https://arxiv.org/abs/2307.04305 · HTML: https://mlsanity.com/paper/2307.04305/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [TorchAudio: Building Blocks for Audio and Speech Processing](https://mlsanity.com/paper/2110.15018/) (2021)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times](https://mlsanity.com/paper/2010.01815/) (2020)
- [Sequence-to-Sequence Piano Transcription with Transformers](https://mlsanity.com/paper/2107.09142/) (2021)
- [On the Potential of Simple Framewise Approaches to Piano Transcription](https://mlsanity.com/paper/1612.05153/) (2016)
- [SpecTNT: a Time-Frequency Transformer for Music Audio](https://mlsanity.com/paper/2110.09127/) (2021)
- [Deep Polyphonic ADSR Piano Note Transcription](https://mlsanity.com/paper/1906.09165/) (2019)
- [Adversarial Learning for Improved Onsets and Frames Music Transcription](https://mlsanity.com/paper/1906.08512/) (2019)
- [Polyphonic Piano Transcription Using Autoregressive Multi-State Note Model](https://mlsanity.com/paper/2010.01104/) (2020)
- [The Effect of Spectrogram Reconstruction on Automatic Music Transcription: An Alternative Approach to Improve Transcription Accuracy](https://mlsanity.com/paper/2010.09969/) (2020)
- [towards automatic transcription of polyphonic electric guitar music:a new dataset and a multi-loss transformer model](https://mlsanity.com/paper/2202.09907/) (2022)
- [HPPNet: Modeling the Harmonic Structure and Pitch Invariance in Piano Transcription](https://mlsanity.com/paper/2208.14339/) (2022)
- [Exploring Transformer's potential on automatic piano transcription](https://mlsanity.com/paper/2204.03898/) (2022)

## Built on by (most cited first, 22 of 22)

- [A Data-Driven Analysis of Robust Automatic Piano Transcription](https://mlsanity.com/paper/2402.01424/) (2024)
- [Scoring Time Intervals using Non-Hierarchical Transformer For Automatic Piano Transcription](https://mlsanity.com/paper/2404.09466/) (2024)
- [YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation](https://mlsanity.com/paper/2407.04822/) (2024)
- [Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling](https://mlsanity.com/paper/2504.15071/) (2025)
- [OpenMU: Your Swiss Army Knife for Music Understanding](https://mlsanity.com/paper/2410.15573/) (2024)
- [Scaling Self-Supervised Representation Learning for Symbolic Piano Performance](https://mlsanity.com/paper/2506.23869/) (2025)
- [Music Foundation Model as Generic Booster for Music Downstream Tasks](https://mlsanity.com/paper/2411.01135/) (2024)
- [Toward Fully Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2402.15569/) (2024)
- [PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance](https://mlsanity.com/paper/2406.09326/) (2024)
- [Quantifying the Corpus Bias Problem in Automatic Music Transcription Systems](https://mlsanity.com/paper/2408.04737/) (2024)
- [D3RM: A Discrete Denoising Diffusion Refinement Model for Piano Transcription](https://mlsanity.com/paper/2501.05068/) (2025)
- [Annotation-free Automatic Music Transcription with Scalable Synthetic Data and Adversarial Domain Confusion](https://mlsanity.com/paper/2312.10402/) (2023)
- [Sound and Music Biases in Deep Music Transcription Models: A Systematic Analysis](https://mlsanity.com/paper/2512.14602/) (2025)
- [AMT-APC: Automatic Piano Cover by Fine-Tuning an Automatic Music Transcription Model](https://mlsanity.com/paper/2409.14086/) (2024)
- [Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoders](https://mlsanity.com/paper/2501.03038/) (2025)
- [Etude: Piano Cover Generation with a Three-Stage Approach -- Extract, strucTUralize, and DEcode](https://mlsanity.com/paper/2509.16522/) (2025)
- [Noise-to-Notes: Diffusion-based Generation and Refinement for Automatic Drum Transcription](https://mlsanity.com/paper/2509.21739/) (2025)
- [VioPTT: Violin Technique-Aware Transcription from Synthetic Data Augmentation](https://mlsanity.com/paper/2509.23759/) (2025)
- [RUMAA: Repeat-Aware Unified Music Audio Analysis for Score-Performance Alignment, Transcription, and Mistake Detection](https://mlsanity.com/paper/2507.12175/) (2025)
- [Disentangling Score Content and Performance Style for Joint Piano Rendering and Transcription](https://mlsanity.com/paper/2509.23878/) (2025)
- [Break-the-Beat! Controllable MIDI-to-Drum Audio Synthesis](https://mlsanity.com/paper/2605.14555/) (2026)
- [Learning to Predict Performance-induced Emotion Differences in Classical Piano Music](https://mlsanity.com/paper/2607.28876/) (2026)

## Abstract (arXiv)

Taking long-term spectral and temporal dependencies into account is essential for automatic piano transcription. This is especially helpful when determining the precise onset and offset for each note in the polyphonic piano content. In this case, we may rely on the capability of self-attention mechanism in Transformers to capture these long-term dependencies in the frequency and time axes. In this work, we propose hFT-Transformer, which is an automatic music transcription method that uses a two-level hierarchical frequency-time Transformer architecture. The first hierarchy includes a convolutional block in the time axis, a Transformer encoder in the frequency axis, and a Transformer decoder that converts the dimension in the frequency axis. The output is then fed into the second hierarchy which consists of another Transformer encoder in the time axis. We evaluated our method with the widely used MAPS and MAESTRO v3.0.0 datasets, and it demonstrated state-of-the-art performance on all the F1-scores of the metrics among Frame, Note, Note with Offset, and Note with Offset and Velocity estimations.