# Unsupervised Harmonic Parameter Estimation Using Differentiable DSP and Spectral Optimal Transport

arXiv: https://arxiv.org/abs/2312.14507 · HTML: https://mlsanity.com/paper/2312.14507/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 22 Dec 2023.

## Builds on (most cited first, 17 of 17)

- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Neural source-filter waveform models for statistical parametric speech synthesis](https://mlsanity.com/paper/1904.12088/) (2019)
- [SPICE: Self-supervised Pitch Estimation](https://mlsanity.com/paper/1910.11664/) (2019)
- [Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks](https://mlsanity.com/paper/1808.06719/) (2018)
- [A Review of Differentiable Digital Signal Processing for Music & Speech Synthesis](https://mlsanity.com/paper/2308.15422/) (2023)
- [The Wasserstein-Fourier Distance for Stationary Time Series](https://mlsanity.com/paper/1912.05509/) (2019)
- [PESTO: Pitch Estimation with Self-supervised Transposition-equivariant Objective](https://mlsanity.com/paper/2309.02265/) (2023)
- [I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch](https://mlsanity.com/paper/2012.04572/) (2020)
- [Cross-domain Neural Pitch and Periodicity Estimation](https://mlsanity.com/paper/2301.12258/) (2023)
- [Sinusoidal Frequency Estimation by Gradient Descent](https://mlsanity.com/paper/2210.14476/) (2022)
- [Optimal spectral transportation with application to music transcription](https://mlsanity.com/paper/1609.09799/) (2016)
- [DDSP-based Singing Vocoders: A New Subtractive-based Synthesizer and A Comprehensive Evaluation](https://mlsanity.com/paper/2208.04756/) (2022)
- [Blind Source Separation with Optimal Transport Non-negative Matrix Factorization](https://mlsanity.com/paper/1802.05429/) (2018)
- [Audio Transport: A Generalized Portamento via Optimal Transport](https://mlsanity.com/paper/1906.06763/) (2019)

## Built on by (most cited first, 8 of 8)

- [Audio synthesizer inversion in symmetric parameter spaces with approximately equivariant flow matching](https://mlsanity.com/paper/2506.07199/) (2025)
- [Toward Fully Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2402.15569/) (2024)
- [Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation](https://mlsanity.com/paper/2407.05516/) (2024)
- [The Inverse Drum Machine: Source Separation Through Joint Transcription and Analysis-by-Synthesis](https://mlsanity.com/paper/2505.03337/) (2025)
- [Fast Differentiable Modal Simulation of Non-linear Strings, Membranes, and Plates](https://mlsanity.com/paper/2505.05940/) (2025)
- [PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective](https://mlsanity.com/paper/2508.01488/) (2025)
- [Translation-Equivariant Self-Supervised Learning for Pitch Estimation with Optimal Transport](https://mlsanity.com/paper/2508.01493/) (2025)
- [Time-Varying Audio Effect Modeling by End-to-End Adversarial Training](https://mlsanity.com/paper/2512.15313/) (2025)

## Abstract (arXiv)

In neural audio signal processing, pitch conditioning has been used to enhance the performance of synthesizers. However, jointly training pitch estimators and synthesizers is a challenge when using standard audio-to-audio reconstruction loss, leading to reliance on external pitch trackers. To address this issue, we propose using a spectral loss function inspired by optimal transportation theory that minimizes the displacement of spectral energy. We validate this approach through an unsupervised autoencoding task that fits a harmonic template to harmonic signals. We jointly estimate the fundamental frequency and amplitudes of harmonics using a lightweight encoder and reconstruct the signals using a differentiable harmonic synthesizer. The proposed approach offers a promising direction for improving unsupervised parameter estimation in neural audio applications.