mlsanityData to 7 Oct 2026

PerMod: Perceptually Grounded Voice Modification with Latent Diffusion Models

arXiv 2312.08494 ↗First posted 13 Dec 2023cs.SD cs.LG eess.AScited by 1

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Perceptual modification of voice is an elusive goal. While non-experts can modify an image or sentence perceptually with available tools, it is not clear how to similarly modify speech along perceptual axes. Voice conversion does make it possible to convert one voice to another, but these modifications are handled by black box models, and the specifics of what perceptual qualities to modify and how to modify them are unclear. Towards allowing greater perceptual control over voice, we introduce PerMod, a conditional latent diffusion model that takes in an input voice and a perceptual qualities vector, and produces a voice with the matching perceptual qualities. Unlike prior work, PerMod generates a new voice corresponding to specific perceptual modifications. Evaluating perceptual quality vectors with RMSE from both human and predicted labels, we demonstrate that PerMod produces voices with the desired perceptual qualities for typical voices, but performs poorly on atypical voices.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 15 of its 29 references that are in the map. First posted on arXiv 13 Dec 2023.

Builds on

15 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models3 Apr
  2. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan

Other references · 13

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  9. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 514
  10. 2021Robust Compressed Sensing MRI with Deep Generative Priorscited by 284
  11. 2022Emotion Intensity and its Control for Emotional Voice Conversioncited by 35
  12. 2022Robust Disentangled Variational Speech Representation Learning for Zero-shot Voice Conversioncited by 28
1 more
  1. 2023Towards an Interpretable Representation of Speaker Identity via Perceptual Voice Qualitiescited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Prompt-guided Precise Audio Editing with Diffusion Modelscited by 18

Similar, not cited

Close in wording, no citation either way

  1. 2024Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement
  2. 2026Adapting Diffusion-Based Music Synthesis to Speech and Singing Voice Conversion
  3. 2025Text-Driven Voice Conversion via Latent State-Space Modeling ↗
  4. 2026VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation
  5. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  6. 2023Creating Personalized Synthetic Voices from Post-Glossectomy Speech with Guided Diffusion Models
  7. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
  8. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers