mlsanityData to 7 Oct 2026

Creating Personalized Synthetic Voices from Post-Glossectomy Speech with Guided Diffusion Models

arXiv 2305.17436 ↗First posted 27 May 2023eess.AScited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

This paper is about developing personalized speech synthesis systems with recordings of mildly impaired speech. In particular, we consider consonant and vowel alterations resulted from partial glossectomy, the surgical removal of part of the tongue. The aim is to restore articulation in the synthesized speech and maximally preserve the target speaker's individuality. We propose to tackle the problem with guided diffusion models. Specifically, a diffusion-based speech synthesis model is trained on original recordings, to capture and preserve the target speaker's original articulation style. When using the model for inference, a separately trained phone classifier will guide the synthesis process towards proper articulation. Objective and subjective evaluation results show that the proposed method substantially improves articulation in the synthesized speech over original recordings, and preserves more of the target speaker's individuality than a voice conversion baseline.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 13 of its 29 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

13 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance23 Nov
  2. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 11

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  8. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  9. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  10. 2019Jasper: An End-to-End Convolutional Neural Acoustic Modelcited by 75
  11. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Diffusion-Based Mel-Spectrogram Enhancement for Personalized Speech Synthesis with Found Data
  2. 2025DiffusionTalker: Efficient and Compact Speech-Driven 3D Talking Head via Personalizer-Guided Distillation
  3. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  4. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme
  5. 2023PerMod: Perceptually Grounded Voice Modification with Latent Diffusion Models
  6. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis
  7. 2025DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model
  8. 2024An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data