# On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis

arXiv: https://arxiv.org/abs/2110.01147 · HTML: https://mlsanity.com/paper/2110.01147/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 4 Oct 2021.

## Builds on (most cited first, 28 of 28)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech](https://mlsanity.com/paper/1807.07281/) (2018)
- [WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis](https://mlsanity.com/paper/2106.09660/) (2021)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [WaveFlow: A Compact Flow-based Model for Raw Audio](https://mlsanity.com/paper/1912.01219/) (2019)
- [Optimizing Speech Recognition For The Edge](https://mlsanity.com/paper/1909.12408/) (2019)
- [LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search](https://mlsanity.com/paper/2102.04040/) (2021)
- [Dynamic Sparsity Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2005.10627/) (2020)
- [TTS-by-TTS: TTS-driven Data Augmentation for Fast and High-Quality Speech Synthesis](https://mlsanity.com/paper/2010.13421/) (2020)
- [EfficientTTS: An Efficient and High-Quality Text-to-Speech Architecture](https://mlsanity.com/paper/2012.03500/) (2020)
- [Neural Waveshaping Synthesis](https://mlsanity.com/paper/2107.05050/) (2021)
- [SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis](https://mlsanity.com/paper/2001.05685/) (2020)

## Built on by (most cited first, 3 of 3)

- [Nix-TTS: Lightweight and End-to-End Text-to-Speech via Module-wise Distillation](https://mlsanity.com/paper/2203.15643/) (2022)
- [EPIC TTS Models: Empirical Pruning Investigations Characterizing Text-To-Speech Models](https://mlsanity.com/paper/2209.10890/) (2022)
- [Paradee: Distilling Kokoro-82M into an 8M-Parameter Single-Voice Text-to-Speech Model](https://mlsanity.com/paper/2610.06817/) (2026)

## Abstract (arXiv)

Are end-to-end text-to-speech (TTS) models over-parametrized? To what extent can these models be pruned, and what happens to their synthesis capabilities? This work serves as a starting point to explore pruning both spectrogram prediction networks and vocoders. We thoroughly investigate the tradeoffs between sparsity and its subsequent effects on synthetic speech. Additionally, we explored several aspects of TTS pruning: amount of finetuning data versus sparsity, TTS-Augmentation to utilize unspoken text, and combining knowledge distillation and pruning. Our findings suggest that not only are end-to-end TTS models highly prunable, but also, perhaps surprisingly, pruned TTS models can produce synthetic speech with equal or higher naturalness and intelligibility, with similar prosody. All of our experiments are conducted on publicly available models, and findings in this work are backed by large-scale subjective tests and objective measures. Code and 200 pruned models are made available to facilitate future research on efficiency in TTS.