mlsanityData to 14 Oct 2026

PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modeling

arXiv 2306.07489 ↗First posted 13 Jun 2023eess.AS cs.AI cs.SDcited by 5

Not on the map. Why?

Abstract (arXiv)

Although text-to-speech (TTS) systems have significantly improved, most TTS systems still have limitations in synthesizing speech with appropriate phrasing. For natural speech synthesis, it is important to synthesize the speech with a phrasing structure that groups words into phrases based on semantic information. In this paper, we propose PuaseSpeech, a speech synthesis system with a pre-trained language model and pause-based prosody modeling. First, we introduce a phrasing structure encoder that utilizes a context representation from the pre-trained language model. In the phrasing structure encoder, we extract a speaker-dependent syntactic representation from the context representation and then predict a pause sequence that separates the input text into phrases. Furthermore, we introduce a pause-based word encoder to model word-level prosody based on pause sequence. Experimental results show PauseSpeech outperforms previous models in terms of naturalness. Furthermore, in terms of objective evaluations, we can observe that our proposed methods help the model decrease the distance between ground-truth and synthesized speech. Audio samples are available at https://jisang93.github.io/pausespeech-demo/.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

33 of 46 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2015Variational Inference with Normalizing Flowscited by 2,170
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2018Self-Attention with Relative Position Representationscited by 1,120
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  12. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
21 more
  1. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  2. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  3. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  4. 2019Assessing BERT's Syntactic Abilitiescited by 192
  5. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 151
  6. 2020End-to-End Adversarial Text-to-Speechcited by 77
  7. 2021PortaSpeech: Portable and High-Quality Generative Text-to-Speechcited by 47
  8. 2022Revisiting Over-Smoothness in Text to Speechcited by 32
  9. 2020Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesiscited by 28
  10. 2020Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesiscited by 27
  11. 2021Fre-GAN: Adversarial Frequency-consistent Audio Synthesiscited by 24
  12. 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speechcited by 22
  13. 2018Sequence-to-Sequence Acoustic Modeling for Voice Conversioncited by 20
  14. 2022SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speechcited by 18
  15. 2020Adversarially Trained Multi-Singer Sequence-To-Sequence Singing Synthesizercited by 12
  16. 2021Phrase break prediction with bidirectional encoder representations in Japanese text-to-speech synthesiscited by 12
  17. 2021Emphasis control for parallel neural TTScited by 9
  18. 2022Expressive, Variable, and Controllable Duration Modelling in TTScited by 7
  19. 2022Polyphone disambiguation and accent prediction using pre-trained language models in Japanese TTS front-endcited by 5
  20. 2022Simple and Effective Multi-sentence TTS with Expressive and Coherent Prosodycited by 5
  21. 2023Duration-aware pause insertion using pre-trained language model for multi-speaker text-to-speechcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 2.

5 papers.

  1. 2023DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Trainingcited by 16
  2. 2025Speculative End-Turn Detector for Efficient Speech Chatbot Assistantcited by 6
  3. 2024ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesiscited by 2
  4. 2026Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interactioncited by 1
  5. 2026LLM-Based Multi-Reference Evaluation for Efficient and Robust Assessment of Phrase Break Annotationscited by 0