Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP
Not on the map. Why?
Abstract (arXiv)
Articulatory trajectories like electromagnetic articulography (EMA) provide a low-dimensional representation of the vocal tract filter and have been used as natural, grounded features for speech synthesis. Differentiable digital signal processing (DDSP) is a parameter-efficient framework for audio synthesis. Therefore, integrating low-dimensional EMA features with DDSP can significantly enhance the computational efficiency of speech synthesis. In this paper, we propose a fast, high-quality, and parameter-efficient DDSP articulatory vocoder that can synthesize speech from EMA, F0, and loudness. We incorporate several techniques to solve the harmonics / noise imbalance problem, and add a multi-resolution adversarial loss for better synthesis quality. Our model achieves a transcription word error rate (WER) of 6.67% and a mean opinion score (MOS) of 3.74, with an improvement of 1.63% and 0.16 compared to the state-of-the-art (SOTA) baseline. Our DDSP vocoder is 4.9x faster than the baseline on CPU during inference, and can generate speech of comparable quality with only 0.4M parameters, in contrast to the 9M parameters required by the SOTA.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 4 Sep 2024.
Builds on
27 of 47 references.
References · 27
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
- 2016Least Squares Generative Adversarial Networkscited by 1,000
- 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
- 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
- 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
- 2020DDSP: Differentiable Digital Signal Processingcited by 177
- 2018CREPE: A Convolutional Representation for Pitch Estimationcited by 109
- 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 71
- 2021Chunked Autoregressive GAN for Conditional Waveform Synthesiscited by 38
15 more
- 2021MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modelingcited by 31
- 2022Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speechcited by 30
- 2023A Review of Differentiable Digital Signal Processing for Music & Speech Synthesiscited by 19
- 2023Self-Supervised Models of Speech Infer Universal Articulatory Kinematicscited by 16
- 2021Neural Waveshaping Synthesiscited by 12
- 2022Deep Speech Synthesis from Articulatory Representationscited by 10
- 2020Hierarchical Timbre-Painting and Articulation Generationcited by 9
- 2020Speech Synthesis and Control Using Differentiable DSPcited by 8
- 2021EMA2S: An End-to-End Multimodal Articulatory-to-Speech Systemcited by 5
- 2019Transferring neural speech waveform synthesizers to musical instrument sounds generationcited by 4
- 2023Style Modeling for Multi-Speaker Articulation-to-Speechcited by 4
- 2021Latent Space Explorations of Singing Voice Synthesis using DDSPcited by 3
- 2022Differentiable WORLD Synthesizer-based Neural Vocoder With Application To End-To-End Audio Style Transfercited by 3
- 2024Ultra-lightweight Neural Differential DSP Vocoder For High Quality Speech Synthesiscited by 2
- 2022Embedding a Differentiable Mel-cepstral Synthesis Filter to a Neural Speech Synthesis Systemcited by 1
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.