# Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions

arXiv: https://arxiv.org/abs/2310.17502 · HTML: https://mlsanity.com/paper/2310.17502/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 54 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines](https://mlsanity.com/paper/2010.11567/) (2020)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [Hi-Fi Multi-Speaker English TTS Dataset](https://mlsanity.com/paper/2104.01497/) (2021)
- [CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages](https://mlsanity.com/paper/1903.11269/) (2019)
- [PortaSpeech: Portable and High-Quality Generative Text-to-Speech](https://mlsanity.com/paper/2109.15166/) (2021)
- [Speaker Anonymization Using X-vector and Neural Waveform Models](https://mlsanity.com/paper/1905.13561/) (2019)
- [AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios](https://mlsanity.com/paper/2204.00436/) (2022)
- [Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech](https://mlsanity.com/paper/2103.04088/) (2021)
- [Attentron: Few-Shot Text-to-Speech Utilizing Attention-Based Variable-Length Embedding](https://mlsanity.com/paper/2005.08484/) (2020)
- [Anonymizing Speech with Generative Adversarial Networks to Preserve Speaker Privacy](https://mlsanity.com/paper/2210.07002/) (2022)
- [Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features](https://mlsanity.com/paper/2203.03191/) (2022)
- [DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021](https://mlsanity.com/paper/2110.12612/) (2021)

## Built on by (most cited first, 1 of 1)

- [LISE : Listenable Interpretable Speaker Embeddings](https://mlsanity.com/paper/2606.21305/) (2026)

## Abstract (arXiv)

Customizing voice and speaking style in a speech synthesis system with intuitive and fine-grained controls is challenging, given that little data with appropriate labels is available. Furthermore, editing an existing human's voice also comes with ethical concerns. In this paper, we propose a method to generate artificial speaker embeddings that cannot be linked to a real human while offering intuitive and fine-grained control over the voice and speaking style of the embeddings, without requiring any labels for speaker or style. The artificial and controllable embeddings can be fed to a speech synthesis system, conditioned on embeddings of real humans during training, without sacrificing privacy during inference.