mlsanityData to 7 Oct 2026

Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units

arXiv 2212.09730 ↗First posted 19 Dec 2022cs.SD cs.CL cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

We introduce DISSC, a novel, lightweight method that converts the rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner. Unlike DISSC, most voice conversion (VC) methods focus primarily on timbre, and ignore people's unique speaking style (prosody). The proposed approach uses a pretrained, self-supervised model for encoding speech to discrete units, which makes it simple, effective, and fast to train. All conversion modules are only trained on reconstruction like tasks, thus suitable for any-to-many VC with no paired data. We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate that DISSC significantly outperforms the evaluated baselines. Code and samples are available at https://pages.cs.huji.ac.il/adiyoss-lab/dissc/.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

30 of 48 references.

References · 30

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  4. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  5. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  6. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  8. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  9. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  10. 2021Direct speech-to-speech translation with discrete unitscited by 148
  11. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  12. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
18 more
  1. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 113
  2. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 87
  3. 2019One-shot Voice Conversion by Separating Speaker and Content Representations with Instance Normalizationcited by 82
  4. 2018StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networkscited by 78
  5. 2020AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalizationcited by 44
  6. 2020Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modelingcited by 42
  7. 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
  8. 2021Improving Zero-shot Voice Style Transfer via Disentangled Representation Learningcited by 32
  9. 2017Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial Networkscited by 30
  10. 2021fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkitcited by 24
  11. 2021S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representationscited by 22
  12. 2020Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representationscited by 18
  13. 2021S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representationscited by 17
  14. 2021Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniquescited by 11
  15. 2022Disentangling Prosody Representations with Unsupervised Speech Reconstructioncited by 8
  16. 2022Audio Language Modeling using Perceptually-Guided Discrete Representationscited by 3
  17. 2022A unified one-shot prosody and speaker conversion system with self-supervised discrete speech unitscited by 3
  18. 2022Investigation into Target Speaking Rate Adaptation for Voice Conversioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 6; 2026: 2.

15 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  2. 2024The VoicePrivacy 2024 Challenge Evaluation Plancited by 39
  3. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  4. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  5. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generationcited by 22
  6. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 13
  7. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 11
  8. 2023Back Translation for Speech-to-text Translation Without Transcriptscited by 9
  9. 2026The Third VoicePrivacy Challenge: Preserving Emotional Expressiveness and Linguistic Content in Voice Anonymizationcited by 8
  10. 2024EmoReg: Directional Latent Vector Modeling for Emotional Intensity Regularization in Diffusion-based Voice Conversioncited by 2
  11. 2025Emotion Recognition and Generation: A Comprehensive Review of Face, Speech, and Text Modalitiescited by 2
  12. 2025NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversioncited by 2
3 more
  1. 2024A Pilot Study of Applying Sequence-to-Sequence Voice Conversion to Evaluate the Intelligibility of L2 Speech Using a Native Speaker's Shadowingscited by 1
  2. 2025GmSLM : Generative Marmoset Spoken Language Modelingcited by 0
  3. 2026On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluationcited by 0