mlsanityData to 7 Oct 2026

Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech

arXiv 2103.04088 ↗First posted 6 Mar 2021eess.AS cs.LG cs.SDcited by 37

Not on the map. Why?

Abstract (arXiv)

The few-shot multi-speaker multi-style voice cloning task is to synthesize utterances with voice and speaking style similar to a reference speaker given only a few reference samples. In this work, we investigate different speaker representations and proposed to integrate pretrained and learnable speaker representations. Among different types of embeddings, the embedding pretrained by voice conversion achieves the best performance. The FastSpeech 2 model combined with both pretrained and learnable speaker representations shows great generalization ability on few-shot speakers and achieved 2nd place in the one-shot track of the ICASSP 2021 M2VoC challenge.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 6 Mar 2021.

Builds on

15 of 21 references.

References · 15

  1. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  2. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  3. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  4. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  5. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  6. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  7. 2018Neural Voice Cloning with a Few Samplescited by 100
  8. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  9. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  10. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  11. 2018Sample Efficient Adaptive Text-to-Speechcited by 45
  12. 2020Controllable Emotion Transfer For End-to-End Speech Synthesiscited by 32
3 more
  1. 2019End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Trainingcited by 19
  2. 2020From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraintcited by 14
  3. 2019Multi-Speaker End-to-End Speech Synthesiscited by 2

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 8; 2023: 14; 2024: 7; 2025: 5.

37 papers.

Sort:

By area:

  1. 2021A Survey on Neural Speech Synthesiscited by 182
  2. 2022AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarioscited by 41
  3. 2022M^4I: Multi-modal Models Membership Inferencecited by 30
  4. 2021Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speechcited by 22
  5. 2021Speaker Generationcited by 13
  6. 2022Deep Performer: Score-to-Audio Music Performance Synthesiscited by 12
  7. 2022Synthesizing Dysarthric Speech Using Multi-talker TTS for Dysarthric Speech Recognitioncited by 10
  8. 2023Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation modelcited by 10
  9. 2024HLTCOE JHU Submission to the Voice Privacy Challenge 2024cited by 10
  10. 2022Evaluating and reducing the distance between synthetic and real speech distributionscited by 8
  11. 2022Unsupervised TTS Acoustic Modeling for TTS with Conditional Disentangled Sequential VAEcited by 7
  12. 2023Accurate synthesis of Dysarthric Speech for ASR data augmentationcited by 7
25 more
  1. 2024USAT: A Universal Speaker-Adaptive Text-to-Speech Approachcited by 6
  2. 2022ECAPA-TDNN for Multi-speaker Text-to-speech Synthesiscited by 5
  3. 2023Generalizable Zero-Shot Speaker Adaptive Speech Synthesis with Disentangled Representationscited by 5
  4. 2024Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapterscited by 4
  5. 2025Voice Cloning: Comprehensive Surveycited by 4
  6. 2023Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speechcited by 3
  7. 2024Human-Centered LLM-Agent User Interface: A Position Papercited by 3
  8. 2025Voice Impression Control in Zero-Shot TTScited by 3
  9. 2025Analyzing and Improving Speaker Similarity Assessment for Speech Synthesiscited by 3
  10. 2025VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learningcited by 3
  11. 2022Towards MOOCs for Lipreading: Using Synthetic Talking Heads to Train Humans in Lipreading at Scalecited by 2
  12. 2023Towards Robust FastSpeech 2 by Modelling Residual Multimodalitycited by 2
  13. 2023Strategies in Transfer Learning for Low-Resource Speech Synthesis: Phone Mapping, Features Input, and Source Language Selectioncited by 2
  14. 2024Lightweight Zero-shot Text-to-Speech with Mixture of Adapterscited by 2
  15. 2025Multi-interaction TTS toward professional recording reproductioncited by 2
  16. 2023Adversarial Training of Denoising Diffusion Model Using Dual Discriminators for High-Fidelity Multi-Speaker TTScited by 1
  17. 2023Multi-GradSpeech: Towards Diffusion-based Multi-Speaker Text-to-speech Using Consistent Diffusion Modelscited by 1
  18. 2023Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directionscited by 1
  19. 2024Roadmap towards Superhuman Speech Understanding using Large Language Modelscited by 1
  20. 2023MParrotTTS: Multilingual Multi-speaker Text to Speech Synthesis in Low Resource Settingcited by 0
  21. 2023The Effects of Input Type and Pronunciation Dictionary Usage in Transfer Learning for Low-Resource Text-to-Speechcited by 0
  22. 2023An analysis on the effects of speaker embedding choice in non auto-regressive TTScited by 0
  23. 2023Combining EEG and NLP Features for Predicting Students' Lecture Comprehension using Ensemble Classificationcited by 0
  24. 2023Code-Mixed Text to Speech Synthesis under Low-Resource Constraintscited by 0
  25. 2024Making Social Platforms Accessible: Emotion-Aware Speech Generation with Integrated Text Analysiscited by 0