mlsanityData to 7 Oct 2026

Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech

arXiv 2308.14909 ↗First posted 28 Aug 2023cs.SD cs.LG eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

For personalized speech generation, a neural text-to-speech (TTS) model must be successfully implemented with limited data from a target speaker. To this end, the baseline TTS model needs to be amply generalized to out-of-domain data (i.e., target speaker's speech). However, approaches to address this out-of-domain generalization problem in TTS have yet to be thoroughly studied. In this work, we propose an effective pruning method for a transformer known as sparse attention, to improve the TTS model's generalization abilities. In particular, we prune off redundant connections from self-attention layers whose attention weights are below the threshold. To flexibly determine the pruning strength for searching optimal degree of generalization, we also propose a new differentiable pruning method that allows the model to automatically learn the thresholds. Evaluations on zero-shot multi-speaker TTS verify the effectiveness of our method in terms of voice quality and speaker similarity.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 28 references that are in the map. First posted on arXiv 28 Aug 2023.

Builds on

26 of 28 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  7. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  8. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  9. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  10. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  11. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  12. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
14 more
  1. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  2. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  3. 2020Sparse Sinkhorn Attentioncited by 250
  4. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  5. 2021Learned Token Pruning for Transformerscited by 138
  6. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  7. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  8. 2020MultiSpeech: Multi-Speaker Text to Speech with Transformercited by 42
  9. 2022AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarioscited by 41
  10. 2021Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speechcited by 37
  11. 2022GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speechcited by 21
  12. 2022nnSpeech: Speaker-Guided Conditional Variational Autoencoder for Zero-shot Multi-speaker Text-to-Speechcited by 10
  13. 2022Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpuscited by 10
  14. 2021Generalizing RNN-Transducer to Out-Domain Audio via Sparse Self-Attention Layerscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Voice Cloning: Comprehensive Surveycited by 4
  2. 2025FMSD-TTS: Few-shot Multi-Speaker Multi-Dialect Text-to-Speech Synthesis for "U-Tsang, Amdo and Kham Speech Dataset Generationcited by 3
  3. 2026WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Modelscited by 0