# EfficientSpeech: An On-Device Text to Speech Model

arXiv: https://arxiv.org/abs/2305.13905 · HTML: https://mlsanity.com/paper/2305.13905/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [The Efficiency Misnomer](https://mlsanity.com/paper/2110.12894/) (2021)
- [PortaSpeech: Portable and High-Quality Generative Text-to-Speech](https://mlsanity.com/paper/2109.15166/) (2021)
- [LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search](https://mlsanity.com/paper/2102.04040/) (2021)
- [On-device neural speech synthesis](https://mlsanity.com/paper/2109.08710/) (2021)
- [Mixer-TTS: non-autoregressive, fast and compact text-to-speech model conditioned on language model embeddings](https://mlsanity.com/paper/2110.03584/) (2021)

## Built on by (most cited first, 4 of 4)

- [DCTTS: Discrete Diffusion Model with Contrastive Learning for Text-to-speech Generation](https://mlsanity.com/paper/2309.06787/) (2023)
- [RU-AI: A Large Multimodal Dataset for Machine-Generated Content Detection](https://mlsanity.com/paper/2406.04906/) (2024)
- [Edge-Based Speech Transcription and Synthesis for Kinyarwanda and Swahili Languages](https://mlsanity.com/paper/2510.16497/) (2025)
- [GrainSpeech: Less Context, More Detail for Compact Speech Synthesis](https://mlsanity.com/paper/2609.18856/) (2026)

## Abstract (arXiv)

State of the art (SOTA) neural text to speech (TTS) models can generate natural-sounding synthetic voices. These models are characterized by large memory footprints and substantial number of operations due to the long-standing focus on speech quality with cloud inference in mind. Neural TTS models are generally not designed to perform standalone speech syntheses on resource-constrained and no Internet access edge devices. In this work, an efficient neural TTS called EfficientSpeech that synthesizes speech on an ARM CPU in real-time is proposed. EfficientSpeech uses a shallow non-autoregressive pyramid-structure transformer forming a U-Network. EfficientSpeech has 266k parameters and consumes 90 MFLOPS only or about 1% of the size and amount of computation in modern compact models such as Mixer-TTS. EfficientSpeech achieves an average mel generation real-time factor of 104.3 on an RPi4. Human evaluation shows only a slight degradation in audio quality as compared to FastSpeech2.