# SQuId: Measuring Speech Naturalness in Many Languages

arXiv: https://arxiv.org/abs/2210.06324 · HTML: https://mlsanity.com/paper/2210.06324/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 47 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [Transfer Learning for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1604.02201/) (2016)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [mSLAM: Massively multilingual joint pre-training for speech and text](https://mlsanity.com/paper/2202.01374/) (2022)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [Generalization Ability of MOS Prediction Networks](https://mlsanity.com/paper/2110.02635/) (2021)
- [Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning](https://mlsanity.com/paper/1907.04448/) (2019)
- [The VoiceMOS Challenge 2022](https://mlsanity.com/paper/2203.11389/) (2022)
- [Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Model](https://mlsanity.com/paper/1712.01541/) (2017)
- [MBNet: MOS Prediction for Synthesized Speech with Mean-Bias Network](https://mlsanity.com/paper/2103.00110/) (2021)
- [Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Features](https://mlsanity.com/paper/2111.02363/) (2021)
- [LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech](https://mlsanity.com/paper/2110.09103/) (2021)
- [AutoMOS: Learning a non-intrusive assessor of naturalness-of-speech](https://mlsanity.com/paper/1611.09207/) (2016)
- [Utilizing Self-supervised Representations for MOS Prediction](https://mlsanity.com/paper/2104.03017/) (2021)
- [Multilingual Byte2Speech Models for Scalable Low-resource Speech Synthesis](https://mlsanity.com/paper/2103.03541/) (2021)
- [That Sounds Familiar: an Analysis of Phonetic Representations Transfer Across Languages](https://mlsanity.com/paper/2005.08118/) (2020)
- [How Phonotactics Affect Multilingual and Zero-shot ASR Performance](https://mlsanity.com/paper/2010.12104/) (2020)

## Built on by (most cited first, 10 of 10)

- [E3 TTS: Easy End-to-End Diffusion-based Text to Speech](https://mlsanity.com/paper/2311.00945/) (2023)
- [EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge](https://mlsanity.com/paper/2505.23009/) (2025)
- [Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations](https://mlsanity.com/paper/2303.01664/) (2023)
- [SpeechLMScore: Evaluating speech generation using speech language model](https://mlsanity.com/paper/2212.04559/) (2022)
- [Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech](https://mlsanity.com/paper/2210.15447/) (2022)
- [FLEURS-R: A Restored Multilingual Speech Corpus for Generation Tasks](https://mlsanity.com/paper/2408.06227/) (2024)
- [Translatotron 3: Speech to Speech Translation with Monolingual Data](https://mlsanity.com/paper/2305.17547/) (2023)
- [Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech](https://mlsanity.com/paper/2310.00706/) (2023)
- [Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restoration](https://mlsanity.com/paper/2505.04457/) (2025)
- [Reduce, Reuse, Recycle: Is Perturbed Data better than Other Language augmentation for Low Resource Self-Supervised Speech Models](https://mlsanity.com/paper/2309.12763/) (2023)

## Abstract (arXiv)

Much of text-to-speech research relies on human evaluation, which incurs heavy costs and slows down the development process. The problem is particularly acute in heavily multilingual applications, where recruiting and polling judges can take weeks. We introduce SQuId (Speech Quality Identification), a multilingual naturalness prediction model trained on over a million ratings and tested in 65 locales-the largest effort of this type to date. The main insight is that training one model on many locales consistently outperforms mono-locale baselines. We present our task, the model, and show that it outperforms a competitive baseline based on w2v-BERT and VoiceMOS by 50.0%. We then demonstrate the effectiveness of cross-locale transfer during fine-tuning and highlight its effect on zero-shot locales, i.e., locales for which there is no fine-tuning data. Through a series of analyses, we highlight the role of non-linguistic effects such as sound artifacts in cross-locale transfer. Finally, we present the effect of our design decision, e.g., model size, pre-training diversity, and language rebalancing with several ablation experiments.