PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective
Not on the map. Why?
Abstract (arXiv)
In this paper, we introduce PESTO, a self-supervised learning approach for single-pitch estimation using a Siamese architecture. Our model processes individual frames of a Variable-Q Transform (VQT) and predicts pitch distributions. The neural network is designed to be equivariant to translations, notably thanks to a Toeplitz fully-connected layer. In addition, we construct pitch-shifted pairs by translating and cropping the VQT frames and train our model with a novel class-based transposition-equivariant objective, eliminating the need for annotated data. Thanks to this architecture and training objective, our model achieves remarkable performances while being very lightweight (130k parameters). Evaluations on music and speech datasets (MIR-1K, MDB-stem-synth, and PTDB) demonstrate that PESTO not only outperforms self-supervised baselines but also competes with supervised methods, exhibiting superior cross-dataset generalization. Finally, we enhance PESTO's practical utility by developing a streamable VQT implementation using cached convolutions. Combined with our model's low latency (less than 10 ms) and minimal parameter count, this makes PESTO particularly suitable for real-time applications.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 51 of its 83 references that are in the map. First posted on arXiv 2 Aug 2025.
Builds on
51 of 83 references.
References · 51
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2016Layer Normalizationcited by 5,579
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
38 more
- 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
- 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
- 2020DDSP: Differentiable Digital Signal Processingcited by 177
- 2020Contrastive Learning of General-Purpose Audio Representationscited by 168
- 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 111
- 2018CREPE: A Convolutional Representation for Pitch Estimationcited by 109
- 2021Contrastive Learning of Musical Representationscited by 81
- 2018Explorations in Homeomorphic Variational Auto-Encodingcited by 65
- 2021Equivariant Contrastive Learningcited by 65
- 2019Neural source-filter waveform models for statistical parametric speech synthesiscited by 46
- 2023Self-supervised learning of Split Invariant Equivariant representationscited by 41
- 2022Unsupervised Learning of Group Invariant and Equivariant Representationscited by 38
- 2022BYOL for Audio: Exploring Pre-trained General-purpose Audio Representationscited by 36
- 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 34
- 2018TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfercited by 32
- 2019SPICE: Self-supervised Pitch Estimationcited by 27
- 2022A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimationcited by 27
- 2022EquiMod: An Equivariance Module to Improve Self-Supervised Learningcited by 20
- 2023A Review of Differentiable Digital Signal Processing for Music & Speech Synthesiscited by 19
- 2023PESTO: Pitch Estimation with Self-supervised Transposition-equivariant Objectivecited by 18
- 2023Cross-domain Neural Pitch and Periodicity Estimationcited by 16
- 2022Sinusoidal Frequency Estimation by Gradient Descentcited by 15
- 2024StreamVC: Real-Time Low-Latency Voice Conversioncited by 15
- 2022Equivariant Self-Supervision for Musical Tempo Estimationcited by 9
- 2020Speech Synthesis and Control Using Differentiable DSPcited by 8
- 2023Unsupervised Harmonic Parameter Estimation Using Differentiable DSP and Spectral Optimal Transportcited by 8
- 2024STONE: Self-supervised Tonality Estimatorcited by 8
- 2021DEEPF0: End-To-End Fundamental Frequency Estimation for Music and Speech Signalscited by 7
- 2022Streamable Neural Audio Synthesis With Non-Causal Convolutionscited by 6
- 2022Value Function Decomposition for Iterative Design of Reinforcement Learning Agentscited by 6
- 2024The Sound of Water: Inferring Physical Properties from Pouring Liquidscited by 5
- 2022Playing Technique Detection by Fusing Note Onset Information in Guzheng Performancecited by 4
- 2022Audio Barlow Twins: Self-Supervised Audio Representation Learningcited by 4
- 2024ISPA: Inter-Species Phonetic Alphabet for Transcribing Animal Soundscited by 4
- 2021Latent Space Explorations of Singing Voice Synthesis using DDSPcited by 3
- 2021Real-time Timbre Transfer and Sound Synthesis using DDSPcited by 3
- 2023Finding Tori: Self-supervised Learning for Analyzing Korean Folk Songcited by 3
- 2023Tempo vs. Pitch: understanding self-supervised tempo estimationcited by 2
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.