# Toward Fully Self-Supervised Multi-Pitch Estimation

arXiv: https://arxiv.org/abs/2402.15569 · HTML: https://mlsanity.com/paper/2402.15569/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 99 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Builds on (most cited first, 30 of 39)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2103.09410/) (2021)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications](https://mlsanity.com/paper/1612.08727/) (2016)
- [Sequence-to-Sequence Piano Transcription with Transformers](https://mlsanity.com/paper/2107.09142/) (2021)
- [Supervised and Unsupervised Learning of Audio Representations for Music Understanding](https://mlsanity.com/paper/2210.03799/) (2022)
- [A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation](https://mlsanity.com/paper/2203.09893/) (2022)
- [SPICE: Self-supervised Pitch Estimation](https://mlsanity.com/paper/1910.11664/) (2019)
- [Unaligned Supervision For Automatic Music Transcription in The Wild](https://mlsanity.com/paper/2204.13668/) (2022)
- [Automatic Piano Transcription with Hierarchical Frequency-Time Transformer](https://mlsanity.com/paper/2307.04305/) (2023)
- [ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data](https://mlsanity.com/paper/2107.04954/) (2021)
- [Simultaneous Separation and Transcription of Mixtures with Multiple Polyphonic and Percussive Instruments](https://mlsanity.com/paper/1910.12621/) (2019)
- [PESTO: Pitch Estimation with Self-supervised Transposition-equivariant Objective](https://mlsanity.com/paper/2309.02265/) (2023)
- [Cross-domain Neural Pitch and Periodicity Estimation](https://mlsanity.com/paper/2301.12258/) (2023)
- [Sinusoidal Frequency Estimation by Gradient Descent](https://mlsanity.com/paper/2210.14476/) (2022)
- [Multitrack Music Transcription with a Time-Frequency Perceiver](https://mlsanity.com/paper/2306.10785/) (2023)
- [The Effect of Spectrogram Reconstruction on Automatic Music Transcription: An Alternative Approach to Improve Transcription Accuracy](https://mlsanity.com/paper/2010.09969/) (2020)
- [Transcription Is All You Need: Learning to Separate Musical Mixtures with Score as Supervision](https://mlsanity.com/paper/2010.11904/) (2020)
- [Unsupervised Harmonic Parameter Estimation Using Differentiable DSP and Spectral Optimal Transport](https://mlsanity.com/paper/2312.14507/) (2023)
- [HPPNet: Modeling the Harmonic Structure and Pitch Invariance in Piano Transcription](https://mlsanity.com/paper/2208.14339/) (2022)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [Singer Identity Representation Learning using Self-Supervised Techniques](https://mlsanity.com/paper/2401.05064/) (2024)

## Built on by (most cited first, 5 of 5)

- [Music102: An D12-equivariant transformer for chord progression accompaniment](https://mlsanity.com/paper/2410.18151/) (2024)
- [Investigating an Overfitting and Degeneration Phenomenon in Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2506.23371/) (2025)
- [Translation-Equivariant Self-Supervised Learning for Pitch Estimation with Optimal Transport](https://mlsanity.com/paper/2508.01493/) (2025)
- [MIDI-RAE-JEPA: Hierarchical Representation Learning and Generation for Symbolic Music](https://mlsanity.com/paper/2607.14537/) (2026)
- [CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representations](https://mlsanity.com/paper/2608.30974/) (2026)

## Abstract (arXiv)

Multi-pitch estimation is a decades-long research problem involving the detection of pitch activity associated with concurrent musical events within multi-instrument mixtures. Supervised learning techniques have demonstrated solid performance on more narrow characterizations of the task, but suffer from limitations concerning the shortage of large-scale and diverse polyphonic music datasets with multi-pitch annotations. We present a suite of self-supervised learning objectives for multi-pitch estimation, which encourage the concentration of support around harmonics, invariance to timbral transformations, and equivariance to geometric transformations. These objectives are sufficient to train an entirely convolutional autoencoder to produce multi-pitch salience-grams directly, without any fine-tuning. Despite training exclusively on a collection of synthetic single-note audio samples, our fully self-supervised framework generalizes to polyphonic music mixtures, and achieves performance comparable to supervised models trained on conventional multi-pitch datasets.