mlsanityData to 14 Oct 2026

Rhythm Modeling for Voice Conversion

arXiv 2307.06040 ↗First posted 12 Jul 2023eess.AS cs.LG cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

Voice conversion aims to transform source speech into a different target voice. However, typical voice conversion systems do not account for rhythm, which is an important factor in the perception of speaker identity. To bridge this gap, we introduce Urhythmic-an unsupervised method for rhythm conversion that does not require parallel data or text transcriptions. Using self-supervised representations, we first divide source audio into segments approximating sonorants, obstruents, and silences. Then we model rhythm by estimating speaking rate or the duration distribution of each segment type. Finally, we match the target speaking rate or rhythm by time-stretching the speech segments. Experiments show that Urhythmic outperforms existing unsupervised methods in terms of quality and prosody. Code and checkpoints: https://github.com/bshall/urhythmic. Audio demo page: https://ubisoft-laforge.github.io/speech/urhythmic.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 18 of its 44 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

18 of 44 references.

References · 18

  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  4. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  5. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  6. 2020Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversioncited by 75
  7. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 61
  8. 2020Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challengecited by 55
  9. 2020Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modelingcited by 42
  10. 2020Towards unsupervised phone and word segmentation using self-supervised vector-quantized neural networkscited by 24
  11. 2018AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation Mechanismscited by 22
  12. 2018Sequence-to-Sequence Acoustic Modeling for Voice Conversioncited by 20
6 more
  1. 2020Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissionscited by 19
  2. 2019Towards Fine-Grained Prosody Control for Voice Conversioncited by 7
  3. 2022ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Speedcited by 7
  4. 2021Global Rhythm Style Transfer Without Text Transcriptionscited by 6
  5. 2023DuTa-VC: A Duration-aware Typical-to-atypical Voice Conversion Approach with Diffusion Probabilistic Modelcited by 6
  6. 2022Investigation into Target Speaking Rate Adaptation for Voice Conversioncited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 4.

6 papers.

  1. 2025Unsupervised Rhythm and Voice Conversion of Dysarthric to Healthy Speech for ASRcited by 5
  2. 2025Children's Voice Privacy: First Steps And Emerging Challengescited by 5
  3. 2024Do Discrete Self-Supervised Representations of Speech Capture Tone Distinctions?cited by 3
  4. 2025Analyzing and Improving Speaker Similarity Assessment for Speech Synthesiscited by 3
  5. 2024D-CAPTCHA++: A Study of Resilience of Deepfake CAPTCHA under Transferable Imperceptible Adversarial Attackcited by 1
  6. 2025Unsupervised Rhythm and Voice Conversion to Improve ASR on Dysarthric Speechcited by 1