mlsanityData to 7 Oct 2026

UnitSpeech: Speaker-adaptive Speech Synthesis with Untranscribed Data

arXiv 2306.16083 ↗First posted 28 Jun 2023cs.SD eess.AScited by 15

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We propose UnitSpeech, a speaker-adaptive speech synthesis method that fine-tunes a diffusion-based text-to-speech (TTS) model using minimal untranscribed data. To achieve this, we use the self-supervised unit representation as a pseudo transcript and integrate the unit encoder into the pre-trained TTS model. We train the unit encoder to provide speech content to the diffusion-based decoder and then fine-tune the decoder for speaker adaptation to the reference speaker using a single <unit, speech> pair. UnitSpeech performs speech synthesis tasks such as TTS and voice conversion (VC) in a personalized manner without requiring model re-training for each task. UnitSpeech achieves comparable and superior results on personalized TTS and any-to-any VC tasks compared to previous baselines. Our model also shows widespread adaptive performance on real-world data and other tasks that use a unit sequence as input.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 29 of its 33 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

29 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Grad-StyleSpeech: Any-speaker Adaptive Text-to-Speech Synthesis with Diffusion Models17 Nov
  2. 2022Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data30 May
  3. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme28 Sep
  4. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  10. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  11. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  12. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
12 more
  1. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  2. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  3. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  4. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 269
  5. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  6. 2021AdaSpeech: Adaptive Text to Speech for Custom Voicecited by 98
  7. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  8. 2022Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentationcited by 43
  9. 2020Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modelingcited by 42
  10. 2022AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarioscited by 41
  11. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  12. 2021AdaSpeech 2: Adaptive Text to Speech with Untranscribed Datacited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 4; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  2. 2024FairSSD: Understanding Bias in Synthetic Speech Detectorscited by 9
  3. 2024DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignmentcited by 8
  4. 2024VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editingcited by 7
  5. 2024USAT: A Universal Speaker-Adaptive Text-to-Speech Approachcited by 6
  6. 2024DiffSSD: A Diffusion-Based Dataset For Speech Forensicscited by 4
  7. 2025Voice Cloning: Comprehensive Surveycited by 4
  8. 2024Stable-TTS: Stable Speaker-Adaptive Text-to-Speech Synthesis via Prosody Promptingcited by 3
  9. 2025Entropy-based Coarse and Compressed Semantic Speech Representation Learningcited by 2
  10. 2024Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selectioncited by 1
  11. 2024Analytic Study of Text-Free Speech Synthesis for Raw Audio using a Self-Supervised Learning Modelcited by 1
  12. 2025Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matchingcited by 1
3 more
  1. 2023Stylebook: Content-Dependent Speaking Style Modeling for Any-to-Any Voice Conversion using Only Speech Datacited by 0
  2. 2025Optimizing Speech Multi-View Feature Fusion through Conditional Computationcited by 0
  3. 2026A survey of AI-generated voices and their detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Sample-Efficient Diffusion for Text-To-Speech Synthesis
  2. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis
  3. 2025AdaptVC: High Quality Voice Conversion with Adaptive Learning
  4. 2023High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models
  5. 2026CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
  6. 2025ECTSpeech: Enhancing Efficient Speech Synthesis via Easy Consistency Tuning
  7. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance
  8. 2026LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling