# Multiple F0 Estimation in Vocal Ensembles using Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2009.04172 · HTML: https://mlsanity.com/paper/2009.04172/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2020.

## Built on by (most cited first, 6 of 6)

- [The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling](https://mlsanity.com/paper/2209.14458/) (2022)
- [JaCappella Corpus: A Japanese a Cappella Vocal Ensemble Corpus](https://mlsanity.com/paper/2211.16028/) (2022)
- [Unsupervised Music Source Separation Using Differentiable Parametric Source Models](https://mlsanity.com/paper/2201.09592/) (2022)
- [Extract fundamental frequency based on CNN combined with PYIN](https://mlsanity.com/paper/2208.08354/) (2022)
- [Song Data Cleansing for End-to-End Neural Singer Diarization Using Neural Analysis and Synthesis Framework](https://mlsanity.com/paper/2406.16315/) (2024)
- [Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles](https://mlsanity.com/paper/2610.03656/) (2026)

## Abstract (arXiv)

This paper addresses the extraction of multiple F0 values from polyphonic and a cappella vocal performances using convolutional neural networks (CNNs). We address the major challenges of ensemble singing, i.e., all melodic sources are vocals and singers sing in harmony. We build upon an existing architecture to produce a pitch salience function of the input signal, where the harmonic constant-Q transform (HCQT) and its associated phase differentials are used as an input representation. The pitch salience function is subsequently thresholded to obtain a multiple F0 estimation output. For training, we build a dataset that comprises several multi-track datasets of vocal quartets with F0 annotations. This work proposes and evaluates a set of CNNs for this task in diverse scenarios and data configurations, including recordings with additional reverb. Our models outperform a state-of-the-art method intended for the same music genre when evaluated with an increased F0 resolution, as well as a general-purpose method for multi-F0 estimation. We conclude with a discussion on future research directions.