mlsanityData to 14 Oct 2026

Voice Attribute Editing with Text Prompt

arXiv 2404.08857 ↗First posted 13 Apr 2024cs.SD cs.AI eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

Despite recent advancements in speech generation with text prompt providing control over speech style, voice attributes in synthesized speech remain elusive and challenging to control. This paper introduces a novel task: voice attribute editing with text prompt, with the goal of making relative modifications to voice attributes according to the actions described in the text prompt. To solve this task, VoxEditor, an end-to-end generative model, is proposed. In VoxEditor, addressing the insufficiency of text prompt, a Residual Memory (ResMem) block is designed, that efficiently maps voice attributes and these descriptors into the shared feature space. Additionally, the ResMem block is enhanced with a voice attribute degree prediction (VADP) block to align voice attributes with corresponding descriptors, addressing the imprecision of text prompt caused by non-quantitative descriptions of voice attributes. We also establish the open-source VCTK-RVA dataset, which leads the way in manual annotations detailing voice characteristic differences among different speakers. Extensive experiments demonstrate the effectiveness and generalizability of our proposed method in terms of both objective and subjective metrics. The dataset and audio samples are available on the website.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 13 Apr 2024.

Builds on

31 of 56 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  7. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  9. 2014Memory Networkscited by 363
  10. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  11. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 185
  12. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
19 more
  1. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
  2. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 101
  3. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 92
  4. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  5. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 80
  6. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  7. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  8. 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
  9. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 48
  10. 2023PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptionscited by 34
  11. 2021Improving Zero-shot Voice Style Transfer via Disentangled Representation Learningcited by 32
  12. 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 25
  13. 2018Sequence-to-Sequence Acoustic Modeling for Voice Conversioncited by 20
  14. 2023PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Promptscited by 18
  15. 2024Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language Promptcited by 12
  16. 2023Face-Driven Zero-Shot Voice Conversion with Memory-based Face-Voice Alignmentcited by 8
  17. 2023Coco-Nut: Corpus of Japanese Utterance and Voice Characteristics Description for Prompt-based Controlcited by 7
  18. 2024VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editingcited by 7
  19. 2023Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement through Knowledge Distillationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2025Scaling Rich Style-Prompted Text-to-Speech Datasetscited by 19
  2. 2024SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Modelcited by 3
  3. 2025Unispeaker: A Unified Approach for Multimodality-driven Speaker Generationcited by 2
  4. 2025The Voice Timbre Attribute Detection 2025 Challenge Evaluation Plancited by 2
  5. 2025Multi-interaction TTS toward professional recording reproductioncited by 2
  6. 2025Introducing voice timbre attribute detectioncited by 1
  7. 2025The First Voice Timbre Attribute Detection Challengecited by 0
  8. 2026RIVET: Robust Idempotent Voice Attribute Editingcited by 0
  9. 2026Encoder-Decoder Manifold Alignment for Idempotent Generationcited by 0