mlsanityData to 7 Oct 2026

On Prosody Modeling for ASR+TTS based Voice Conversion

arXiv 2107.09477 ↗First posted 20 Jul 2021cs.SD cs.CL eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

In voice conversion (VC), an approach showing promising results in the latest voice conversion challenge (VCC) 2020 is to first use an automatic speech recognition (ASR) model to transcribe the source speech into the underlying linguistic contents; these are then used as input by a text-to-speech (TTS) system to generate the converted speech. Such a paradigm, referred to as ASR+TTS, overlooks the modeling of prosody, which plays an important role in speech naturalness and conversion similarity. Although some researchers have considered transferring prosodic clues from the source speech, there arises a speaker mismatch during training and conversion. To address this issue, in this work, we propose to directly predict prosody from the linguistic representation in a target-speaker-dependent manner, referred to as target text prediction (TTP). We evaluate both methods on the VCC2020 benchmark and consider different linguistic representations. The results demonstrate the effectiveness of TTP in both objective and subjective evaluations.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 48 references that are in the map. First posted on arXiv 20 Jul 2021.

Builds on

29 of 48 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  4. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  5. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  6. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  7. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  8. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  9. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  10. 2020Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversioncited by 75
  11. 2018Learning latent representations for style control and transfer in end-to-end speech synthesiscited by 61
  12. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
17 more
  1. 2019CSS10: A Collection of Single Speaker Speech Datasets for 10 Languagescited by 54
  2. 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
  3. 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
  4. 2018Predicting Expressive Speaking Style From Text In End-To-End Speech Synthesiscited by 31
  5. 2019Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretrainingcited by 29
  6. 2019Fine-grained robust prosody transfer for single-speaker neural text-to-speechcited by 27
  7. 2020An Open source Implementation of ITU-T Recommendation P.808 with Validationcited by 27
  8. 2018Sequence-to-Sequence Acoustic Modeling for Voice Conversioncited by 20
  9. 2020Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissionscited by 19
  10. 2020Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representationscited by 18
  11. 2019Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesiscited by 15
  12. 2020Pretraining Techniques for Sequence-to-Sequence Voice Conversioncited by 13
  13. 2020The Sequence-to-Sequence Baseline for the Voice Conversion Challenge 2020: Cascading ASR and TTScited by 12
  14. 2020Voice Conversion by Cascading Automatic Speech Recognition and Text-to-Speech Synthesis with Prosody Transfercited by 10
  15. 2019Towards Fine-Grained Prosody Control for Voice Conversioncited by 7
  16. 2020Transferring Source Style in Non-Parallel Voice Conversioncited by 7
  17. 2020The Academia Sinica Systems of Voice Conversion for VCC2020cited by 2

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 1.

3 papers.

  1. 2021S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representationscited by 22
  2. 2022A Comparative Study of Self-supervised Speech Representation Based Voice Conversioncited by 9
  3. 2023Automatic Speech Disentanglement for Voice Conversion using Rank Module and Speech Augmentationcited by 0