# An analysis on the effects of speaker embedding choice in non auto-regressive TTS

arXiv: https://arxiv.org/abs/2307.09898 · HTML: https://mlsanity.com/paper/2307.09898/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://mlsanity.com/paper/1705.08947/) (2017)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech](https://mlsanity.com/paper/2103.04088/) (2021)
- [Clova Baseline System for the VoxCeleb Speaker Recognition Challenge 2020](https://mlsanity.com/paper/2009.14153/) (2020)
- [Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data](https://mlsanity.com/paper/2205.15370/) (2022)
- [Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech](https://mlsanity.com/paper/2111.04040/) (2021)
- [Probing the Information Encoded in X-vectors](https://mlsanity.com/paper/1909.06351/) (2019)
- [Low-Resource Multilingual and Zero-Shot Multispeaker TTS](https://mlsanity.com/paper/2210.12223/) (2022)
- [From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint](https://mlsanity.com/paper/2005.04587/) (2020)
- [Residual Information in Deep Speaker Embedding Architectures](https://mlsanity.com/paper/2302.02742/) (2023)

## Abstract (arXiv)

In this paper we introduce a first attempt on understanding how a non-autoregressive factorised multi-speaker speech synthesis architecture exploits the information present in different speaker embedding sets. We analyse if jointly learning the representations, and initialising them from pretrained models determine any quality improvements for target speaker identities. In a separate analysis, we investigate how the different sets of embeddings impact the network's core speech abstraction (i.e. zero conditioned) in terms of speaker identity and representation learning. We show that, regardless of the used set of embeddings and learning strategy, the network can handle various speaker identities equally well, with barely noticeable variations in speech output quality, and that speaker leakage within the core structure of the synthesis system is inevitable in the standard training procedures adopted thus far.