# Lightweight and High-Fidelity End-to-End Text-to-Speech with Multi-Band Generation and Inverse Short-Time Fourier Transform

arXiv: https://arxiv.org/abs/2210.15975 · HTML: https://mlsanity.com/paper/2210.15975/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech](https://mlsanity.com/paper/2005.05106/) (2020)
- [LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search](https://mlsanity.com/paper/2102.04040/) (2021)
- [SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis](https://mlsanity.com/paper/2001.05685/) (2020)
- [DeviceTTS: A Small-Footprint, Fast, Stable Network for On-Device Text-to-Speech](https://mlsanity.com/paper/2010.15311/) (2020)
- [Nix-TTS: Lightweight and End-to-End Text-to-Speech via Module-wise Distillation](https://mlsanity.com/paper/2203.15643/) (2022)

## Built on by (most cited first, 9 of 9)

- [SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesis](https://mlsanity.com/paper/2504.09839/) (2025)
- [Low-latency Real-time Voice Conversion on CPU](https://mlsanity.com/paper/2311.00873/) (2023)
- [BitTTS: Highly Compact Text-to-Speech Using 1.58-bit Quantization and Weight Indexing](https://mlsanity.com/paper/2506.03515/) (2025)
- [Lightweight End-to-end Text-to-speech Synthesis for low resource on-device applications](https://mlsanity.com/paper/2505.07701/) (2025)
- [FNH-TTS: Mixture-of-Experts Duration Modeling for Robust Neural Speech Synthesis](https://mlsanity.com/paper/2508.12001/) (2025)
- [VocalBridge: Latent Diffusion-Bridge Purification for Defeating Perturbation-Based Voiceprint Defenses](https://mlsanity.com/paper/2601.02444/) (2026)
- [Morpho-VITS: Variational Inference with Morphological Modeling for End-to-End Speech Synthesis of a Tonal Bantu Language](https://mlsanity.com/paper/2609.24310/) (2026)
- [BanglaKontho: Closing the Long-Form Gap in Bangla Text-to-Speech](https://mlsanity.com/paper/2609.29146/) (2026)
- [Paradee: Distilling Kokoro-82M into an 8M-Parameter Single-Voice Text-to-Speech Model](https://mlsanity.com/paper/2610.06817/) (2026)

## Abstract (arXiv)

We propose a lightweight end-to-end text-to-speech model using multi-band generation and inverse short-time Fourier transform. Our model is based on VITS, a high-quality end-to-end text-to-speech model, but adopts two changes for more efficient inference: 1) the most computationally expensive component is partially replaced with a simple inverse short-time Fourier transform, and 2) multi-band generation, with fixed or trainable synthesis filters, is used to generate waveforms. Unlike conventional lightweight models, which employ optimization or knowledge distillation separately to train two cascaded components, our method enjoys the full benefits of end-to-end optimization. Experimental results show that our model synthesized speech as natural as that synthesized by VITS, while achieving a real-time factor of 0.066 on an Intel Core i7 CPU, 4.1 times faster than VITS. Moreover, a smaller version of the model significantly outperformed a lightweight baseline model with respect to both naturalness and inference speed. Code and audio samples are available from https://github.com/MasayaKawamura/MB-iSTFT-VITS.