# Onsets and Frames: Dual-Objective Piano Transcription

arXiv: https://arxiv.org/abs/1710.11153 · HTML: https://mlsanity.com/paper/1710.11153/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2017.

## Built on by (most cited first, 30 of 94)

- [Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs](https://mlsanity.com/paper/2101.02402/) (2021)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [General-purpose, long-context autoregressive modeling with Perceiver AR](https://mlsanity.com/paper/2202.07765/) (2022)
- [Sequence-to-Sequence Piano Transcription with Transformers](https://mlsanity.com/paper/2107.09142/) (2021)
- [MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE](https://mlsanity.com/paper/2105.04090/) (2021)
- [Relative Positional Encoding for Transformers with Linear Complexity](https://mlsanity.com/paper/2105.08399/) (2021)
- [Video Background Music Generation: Dataset, Method and Evaluation](https://mlsanity.com/paper/2211.11248/) (2022)
- [A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation](https://mlsanity.com/paper/2203.09893/) (2022)
- [Unaligned Supervision For Automatic Music Transcription in The Wild](https://mlsanity.com/paper/2204.13668/) (2022)
- [Melody transcription via generative pre-training](https://mlsanity.com/paper/2212.01884/) (2022)
- [Automatic Piano Transcription with Hierarchical Frequency-Time Transformer](https://mlsanity.com/paper/2307.04305/) (2023)
- [ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data](https://mlsanity.com/paper/2107.04954/) (2021)
- [A Data-Driven Analysis of Robust Automatic Piano Transcription](https://mlsanity.com/paper/2402.01424/) (2024)
- [High Resolution Guitar Transcription via Domain Adaptation](https://mlsanity.com/paper/2402.15258/) (2024)
- [Scoring Time Intervals using Non-Hierarchical Transformer For Automatic Piano Transcription](https://mlsanity.com/paper/2404.09466/) (2024)
- [DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability](https://mlsanity.com/paper/2210.05148/) (2022)
- [Multitrack Music Transcription with a Time-Frequency Perceiver](https://mlsanity.com/paper/2306.10785/) (2023)
- [YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation](https://mlsanity.com/paper/2407.04822/) (2024)
- [BERT-like Pre-training for Symbolic Piano Music Classification Tasks](https://mlsanity.com/paper/2107.05223/) (2021)
- [Generating music with sentiment using Transformer-GANs](https://mlsanity.com/paper/2212.11134/) (2022)
- [Pop2Piano : Pop Audio-based Piano Cover Generation](https://mlsanity.com/paper/2211.00895/) (2022)
- [Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling](https://mlsanity.com/paper/2504.15071/) (2025)
- [End-to-end Piano Performance-MIDI to Score Conversion with Transformers](https://mlsanity.com/paper/2410.00210/) (2024)
- [Audio-to-symbolic Arrangement via Cross-modal Music Representation Learning](https://mlsanity.com/paper/2112.15110/) (2021)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [An Order-Complexity Aesthetic Assessment Model for Aesthetic-aware Music Recommendation](https://mlsanity.com/paper/2402.08300/) (2024)
- [Music Foundation Model as Generic Booster for Music Downstream Tasks](https://mlsanity.com/paper/2411.01135/) (2024)
- [Unified Cross-modal Translation of Score Images, Symbolic Music, and Performance Audio](https://mlsanity.com/paper/2505.12863/) (2025)
- [Structure-Aware Audio-to-Score Alignment using Progressively Dilated Convolutional Neural Networks](https://mlsanity.com/paper/2102.00382/) (2021)

## Abstract (arXiv)

We advance the state of the art in polyphonic piano music transcription by using a deep convolutional and recurrent neural network which is trained to jointly predict onsets and frames. Our model predicts pitch onset events and then uses those predictions to condition framewise pitch predictions. During inference, we restrict the predictions from the framewise detector by not allowing a new note to start unless the onset detector also agrees that an onset for that pitch is present in the frame. We focus on improving onsets and offsets together instead of either in isolation as we believe this correlates better with human musical perception. Our approach results in over a 100% relative improvement in note F1 score (with offsets) on the MAPS dataset. Furthermore, we extend the model to predict relative velocities of normalized audio which results in more natural-sounding transcriptions.