mlsanityData to 14 Oct 2026

Towards General-Purpose Text-Instruction-Guided Voice Conversion

arXiv 2309.14324 ↗First posted 25 Sep 2023eess.AS cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper introduces a novel voice conversion (VC) model, guided by text instructions such as "articulate slowly with a deep tone" or "speak in a cheerful boyish voice". Unlike traditional methods that rely on reference utterances to determine the attributes of the converted speech, our model adds versatility and specificity to voice conversion. The proposed VC model is a neural codec language model which processes a sequence of discrete codes, resulting in the code sequence of converted speech. It utilizes text instructions as style prompts to modify the prosody and emotional information of the given speech. In contrast to previous approaches, which often rely on employing separate encoders like prosody and content encoders to handle different aspects of the source speech, our model handles various information of speech in an end-to-end manner. Experiments have demonstrated the impressive capabilities of our model in comprehending instructions and delivering reasonable results.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 60 of its 70 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

60 of 70 references.

References · 60

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  4. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  5. 2022High Fidelity Neural Audio Compressioncited by 687
  6. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  7. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  8. 2023Simple and Controllable Music Generationcited by 488
  9. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  10. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  11. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  12. 2022AudioGen: Textually Guided Audio Generationcited by 295
38 more
  1. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  2. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  3. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
  4. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 192
  5. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  6. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  7. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  8. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 115
  9. 2023SoundStorm: Efficient Parallel Audio Generationcited by 114
  10. 2020An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learningcited by 112
  11. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
  12. 2023Textually Pretrained Speech Language Modelscited by 101
  13. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 92
  14. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  15. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  16. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  17. 2018StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networkscited by 78
  18. 2020Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversioncited by 75
  19. 2018The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methodscited by 68
  20. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 61
  21. 2021StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversioncited by 58
  22. 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codeccited by 47
  23. 2019CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversioncited by 46
  24. 2016Voice Conversion from Non-parallel Corpora Using Variational Auto-encodercited by 45
  25. 2017Voice Conversion from Unaligned Corpora using Variational Autoencoding Wasserstein Generative Adversarial Networkscited by 38
  26. 2019StarGAN-VC2: Rethinking Conditional Methods for StarGAN-Based Voice Conversioncited by 37
  27. 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
  28. 2020F0-consistent many-to-many non-parallel voice conversion via conditional autoencodercited by 35
  29. 2023PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptionscited by 34
  30. 2017Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networkscited by 31
  31. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  32. 2020Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Datacited by 26
  33. 2018Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representationscited by 25
  34. 2020CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversioncited by 25
  35. 2021S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representationscited by 23
  36. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  37. 2022SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Taskscited by 19
  38. 2019Singing Voice Conversion with Disentangled Representations of Singer and Vocal Technique Using Variational Autoencoderscited by 18

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Audio-Aware Large Language Models as Judges for Speaking Stylescited by 17
  2. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  3. 2025Gender Bias in Instruction-Guided Speech Synthesis Modelscited by 4
  4. 2025NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversioncited by 2
  5. 2025AudioRole: An Audio Dataset for Character Role-Playing in Large Language Modelscited by 1
  6. 2026Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Modelscited by 0