# High-Fidelity and Low-Latency Universal Neural Vocoder based on Multiband WaveRNN with Data-Driven Linear Prediction for Discrete Waveform Modeling

arXiv: https://arxiv.org/abs/2105.09856 · HTML: https://mlsanity.com/paper/2105.09856/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 20 May 2021.

## Builds on (most cited first, 18 of 18)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [A* Sampling](https://mlsanity.com/paper/1411.0030/) (2014)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion](https://mlsanity.com/paper/2008.12527/) (2020)
- [DurIAN: Duration Informed Attention Network For Multimodal Synthesis](https://mlsanity.com/paper/1909.01700/) (2019)
- [Neural source-filter-based waveform model for statistical parametric speech synthesis](https://mlsanity.com/paper/1810.11946/) (2018)
- [JVS corpus: free Japanese multi-speaker voice corpus](https://mlsanity.com/paper/1908.06248/) (2019)
- [Att-HACK: An Expressive Speech Database with Social Attitudes](https://mlsanity.com/paper/2004.04410/) (2020)
- [FeatherWave: An efficient high-fidelity neural vocoder with multi-band linear prediction](https://mlsanity.com/paper/2005.05551/) (2020)

## Built on by (most cited first, 3 of 3)

- [A Text-to-Speech Pipeline, Evaluation Methodology, and Initial Fine-Tuning Results for Child Speech Synthesis](https://mlsanity.com/paper/2203.11562/) (2022)
- [ALO-VC: Any-to-any Low-latency One-shot Voice Conversion](https://mlsanity.com/paper/2306.01100/) (2023)
- [Improved Child Text-to-Speech Synthesis through Fastpitch-based Transfer Learning](https://mlsanity.com/paper/2311.04313/) (2023)

## Abstract (arXiv)

This paper presents a novel high-fidelity and low-latency universal neural vocoder framework based on multiband WaveRNN with data-driven linear prediction for discrete waveform modeling (MWDLP). MWDLP employs a coarse-fine bit WaveRNN architecture for 10-bit mu-law waveform modeling. A sparse gated recurrent unit with a relatively large size of hidden units is utilized, while the multiband modeling is deployed to achieve real-time low-latency usage. A novel technique for data-driven linear prediction (LP) with discrete waveform modeling is proposed, where the LP coefficients are estimated in a data-driven manner. Moreover, a novel loss function using short-time Fourier transform (STFT) for discrete waveform modeling with Gumbel approximation is also proposed. The experimental results demonstrate that the proposed MWDLP framework generates high-fidelity synthetic speech for seen and unseen speakers and/or language on 300 speakers training data including clean and noisy/reverberant conditions, where the number of training utterances is limited to 60 per speaker, while allowing for real-time low-latency processing using a single core of ~! 2.1--2.7 GHz CPU with ~! 0.57--0.64 real-time factor including input/output and feature extraction.