mlsanityData to 14 Oct 2026

ParaTTS: Learning Linguistic and Prosodic Cross-sentence Information in Paragraph-based TTS

arXiv 2209.06484 ↗First posted 14 Sep 2022cs.SD cs.CL eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Recent advancements in neural end-to-end TTS models have shown high-quality, natural synthesized speech in a conventional sentence-based TTS. However, it is still challenging to reproduce similar high quality when a whole paragraph is considered in TTS, where a large amount of contextual information needs to be considered in building a paragraph-based TTS model. To alleviate the difficulty in training, we propose to model linguistic and prosodic information by considering cross-sentence, embedded structure in training. Three sub-modules, including linguistics-aware, prosody-aware and sentence-position networks, are trained together with a modified Tacotron2. Specifically, to learn the information embedded in a paragraph and the relations among the corresponding component sentences, we utilize linguistics-aware and prosody-aware networks. The information in a paragraph is captured by encoders and the inter-sentence information in a paragraph is learned with multi-head attention mechanisms. The relative sentence position in a paragraph is explicitly exploited by a sentence-position network. Trained on a storytelling audio-book corpus (4.08 hours), recorded by a female Mandarin Chinese speaker, the proposed TTS model demonstrates that it can produce rather natural and good-quality speech paragraph-wise. The cross-sentence contextual information, such as break and prosodic variations between consecutive sentences, can be better predicted and rendered than the sentence-based model. Tested on paragraph texts, of which the lengths are similar to, longer than, or much longer than the typical paragraph length of the training data, the TTS speech produced by the new model is consistently preferred over the sentence-based model in subjective tests and confirmed in objective measures.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 25 of its 57 references that are in the map. First posted on arXiv 14 Sep 2022.

Builds on

25 of 57 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  9. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  10. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  11. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  12. 2015Highway Networkscited by 260
13 more
  1. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  2. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 110
  3. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  4. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
  5. 2020Conversational End-to-End TTS for Voice Agentcited by 37
  6. 2020Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesiscited by 27
  7. 2020CAMP: a Two-Stage Approach to Modelling Prosody in Contextcited by 12
  8. 2019Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven Acoustic Embedding Selectioncited by 11
  9. 2019Exploiting Syntactic Features in a Parsed Tree to Improve End-to-End TTScited by 10
  10. 2020Prosodic Representation Learning and Contextual Sampling for Neural Text-to-Speechcited by 10
  11. 2019Evaluating Long-form Text-to-Speech: Comparing the Ratings of Sentences and Paragraphscited by 8
  12. 2020s-Transformer: Segment-Transformer for Robust Neural Speech Synthesiscited by 5
  13. 2020Using previous acoustic context to improve Text-to-Speech synthesiscited by 3

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2.

5 papers.

  1. 2023DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Trainingcited by 16
  2. 2024MCDubber: Multimodal Context-Aware Expressive Video Dubbingcited by 7
  3. 2023Towards Spontaneous Style Modeling with Semi-supervised Pre-training for Conversational Text-to-Speech Synthesiscited by 4
  4. 2024Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with Multi-Modal Context and Large Language Modelcited by 3
  5. 2023ContextSpeech: Expressive and Efficient Text-to-Speech for Paragraph Readingcited by 2