mlsanityData to 7 Oct 2026

Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters

arXiv 2401.05111 ↗First posted 10 Jan 2024cs.SD cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The zero-shot text-to-speech (TTS) method, based on speaker embeddings extracted from reference speech using self-supervised learning (SSL) speech representations, can reproduce speaker characteristics very accurately. However, this approach suffers from degradation in speech synthesis quality when the reference speech contains noise. In this paper, we propose a noise-robust zero-shot TTS method. We incorporated adapters into the SSL model, which we fine-tuned with the TTS model using noisy reference speech. In addition, to further improve performance, we adopted a speech enhancement (SE) front-end. With these improvements, our proposed SSL-based zero-shot TTS achieved high-quality speech synthesis with noisy reference speech. Through the objective and subjective evaluations, we confirmed that the proposed method is highly robust to noise in reference speech, and effectively works in combination with SE.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

29 of 36 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  10. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  12. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
17 more
  1. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  2. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  3. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  4. 2015End-to-End Text-Dependent Speaker Verificationcited by 49
  5. 2021Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speechcited by 37
  6. 2022Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?cited by 27
  7. 2022JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speechcited by 26
  8. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  9. 2022Boosting Self-Supervised Embeddings for Speech Enhancementcited by 24
  10. 2022End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representationcited by 17
  11. 2020DenoiSpeech: Denoising Text to Speech with Frame-Level Noise Modelingcited by 13
  12. 2022CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Modelscited by 13
  13. 2023Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation modelcited by 10
  14. 2022NoreSpeech: Knowledge Distillation based Conditional Diffusion Model for Noise-robust Expressive TTScited by 8
  15. 2023Downstream Task Agnostic Speech Enhancement with Self-Supervised Representation Losscited by 3
  16. 2022DRSpeech: Degradation-Robust Text-to-Speech Synthesis with Frame-Level and Utterance-Level Acoustic Representation Learningcited by 2
  17. 2022Fine-grained Noise Control for Multispeaker Speech Synthesiscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTScited by 4
  2. 2025Voice Cloning: Comprehensive Surveycited by 4
  3. 2026Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesiscited by 4
  4. 2025AdaptVC: High Quality Voice Conversion with Adaptive Learningcited by 1