# Deep Learning Based Assessment of Synthetic Speech Naturalness

arXiv: https://arxiv.org/abs/2104.11673 · HTML: https://mlsanity.com/paper/2104.11673/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 36 references that are in the map.
- First posted on arXiv 23 Apr 2021.

## Builds on (most cited first, 3 of 3)

- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods](https://mlsanity.com/paper/1804.04262/) (2018)
- [AutoMOS: Learning a non-intrusive assessor of naturalness-of-speech](https://mlsanity.com/paper/1611.09207/) (2016)

## Built on by (most cited first, 30 of 35)

- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling](https://mlsanity.com/paper/2303.03926/) (2023)
- [VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation](https://mlsanity.com/paper/2305.16107/) (2023)
- [The GENEA Challenge 2022: A large evaluation of data-driven co-speech gesture generation](https://mlsanity.com/paper/2208.10441/) (2022)
- [Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022](https://mlsanity.com/paper/2303.08737/) (2023)
- [LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech](https://mlsanity.com/paper/2110.09103/) (2021)
- [Building a Taiwanese Mandarin Spoken Language Model: A First Attempt](https://mlsanity.com/paper/2411.07111/) (2024)
- [Boosting Large Language Model for Speech Synthesis: An Empirical Study](https://mlsanity.com/paper/2401.00246/) (2023)
- [SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesis](https://mlsanity.com/paper/2204.03040/) (2022)
- [EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speech](https://mlsanity.com/paper/2307.00024/) (2023)
- [CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations](https://mlsanity.com/paper/2404.06690/) (2024)
- [Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gap](https://mlsanity.com/paper/2410.16726/) (2024)
- [Evaluating Robustness of Large Audio Language Models to Audio Injection: An Empirical Study](https://mlsanity.com/paper/2505.19598/) (2025)
- [TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation](https://mlsanity.com/paper/2405.17809/) (2024)
- [1000 African Voices: Advancing inclusive multi-speaker multi-accent speech synthesis](https://mlsanity.com/paper/2406.11727/) (2024)
- [Robust One-Shot Singing Voice Conversion](https://mlsanity.com/paper/2210.11096/) (2022)
- [SVSNet: An End-to-end Speaker Voice Similarity Assessment Model](https://mlsanity.com/paper/2107.09392/) (2021)
- [Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representations](https://mlsanity.com/paper/2107.12003/) (2021)
- [Pairwise Evaluation of Accent Similarity in Speech Synthesis](https://mlsanity.com/paper/2505.14410/) (2025)
- [On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition](https://mlsanity.com/paper/2407.21476/) (2024)
- [Learning and controlling the source-filter representation of speech with a variational autoencoder](https://mlsanity.com/paper/2204.07075/) (2022)
- [Wideband Audio Waveform Evaluation Networks: Efficient, Accurate Estimation of Speech Qualities](https://mlsanity.com/paper/2206.13272/) (2022)
- [Investigating Content-Aware Neural Text-To-Speech MOS Prediction Using Prosodic and Linguistic Features](https://mlsanity.com/paper/2211.00342/) (2022)
- [Iteratively Improving Speech Recognition and Voice Conversion](https://mlsanity.com/paper/2305.15055/) (2023)
- [APG-MOS: Auditory Perception Guided-MOS Predictor for Synthetic Speech](https://mlsanity.com/paper/2504.20447/) (2025)
- [The Limits of Reference-Free Speech Quality Metrics as Evaluators and Rewards on Modern Text-to-Speech](https://mlsanity.com/paper/2609.13150/) (2026)
- [Predicting pairwise preferences between TTS audio stimuli using parallel ratings data and anti-symmetric twin neural networks](https://mlsanity.com/paper/2209.11003/) (2022)
- [Cross-modal Face- and Voice-style Transfer](https://mlsanity.com/paper/2302.13838/) (2023)
- [Humane Speech Synthesis through Zero-Shot Emotion and Disfluency Generation](https://mlsanity.com/paper/2404.01339/) (2024)
- [VoiceCloak: A Multi-Dimensional Defense Framework against Unauthorized Diffusion-based Voice Cloning](https://mlsanity.com/paper/2505.12332/) (2025)

## Abstract (arXiv)

In this paper, we present a new objective prediction model for synthetic speech naturalness. It can be used to evaluate Text-To-Speech or Voice Conversion systems and works language independently. The model is trained end-to-end and based on a CNN-LSTM network that previously showed to give good results for speech quality estimation. We trained and tested the model on 16 different datasets, such as from the Blizzard Challenge and the Voice Conversion Challenge. Further, we show that the reliability of deep learning-based naturalness prediction can be improved by transfer learning from speech quality prediction models that are trained on objective POLQA scores. The proposed model is made publicly available and can, for example, be used to evaluate different TTS system configurations.