mlsanityData to 7 Oct 2026

VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing

arXiv 2404.06674 ↗First posted 10 Apr 2024cs.SD cs.AI eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

We present VoiceShop, a novel speech-to-speech framework that can modify multiple attributes of speech, such as age, gender, accent, and speech style, in a single forward pass while preserving the input speaker's timbre. Previous works have been constrained to specialized models that can only edit these attributes individually and suffer from the following pitfalls: the magnitude of the conversion effect is weak, there is no zero-shot capability for out-of-distribution speakers, or the synthesized outputs exhibit undesirable timbre leakage. Our work proposes solutions for each of these issues in a simple modular framework based on a conditional diffusion backbone model with optional normalizing flow-based and sequence-to-sequence speaker attribute-editing modules, whose components can be combined or removed during inference to meet a wide array of tasks without additional model finetuning. Audio samples are available at urlhttps://voiceshopai.github.io.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 68 of its 84 references that are in the map. First posted on arXiv 10 Apr 2024.

Builds on

68 of 84 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2018Neural Ordinary Differential Equationscited by 3,733
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2015Variational Inference with Normalizing Flowscited by 2,161
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  11. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
38 more
  1. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  9. 2022High Fidelity Neural Audio Compressioncited by 673
  10. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  11. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  12. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  13. 2015A Neural Algorithm of Artistic Stylecited by 480
  14. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  15. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 348
  16. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  17. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  18. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  19. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  20. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  21. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  22. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  23. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  24. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  25. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Schemecited by 99
  26. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  27. 2017One Model To Learn Them Allcited by 86
  28. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  29. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  30. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 79
  31. 2024Proactive Detection of Voice Cloning with Localized Watermarkingcited by 67
  32. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 63
  33. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
  34. 2023Detecting Voice Cloning Attacks via Timbre Watermarkingcited by 39
  35. 2022NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesiscited by 38
  36. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversioncited by 38
  37. 2021Citrinet: Closing the Gap between Non-Autoregressive and Autoregressive End-to-End Models for Automatic Speech Recognitioncited by 33
  38. 2023Audio Deepfake Detection with Self-Supervised WavLM and Multi-Fusion Attentive Classifiercited by 26

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 2.

7 papers.

  1. 2024StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matchingcited by 14
  2. 2024Voice Attribute Editing with Text Promptcited by 9
  3. 2025Step-Audio-EditX Technical Reportcited by 6
  4. 2025Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generationcited by 5
  5. 2024Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modelingcited by 1
  6. 2026RIVET: Robust Idempotent Voice Attribute Editingcited by 0
  7. 2026Encoder-Decoder Manifold Alignment for Idempotent Generationcited by 0