mlsanityData to 14 Oct 2026

Automatic Prosody Annotation with Pre-Trained Text-Speech Model

arXiv 2206.07956 ↗First posted 16 Jun 2022cs.SD cs.CL eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Prosodic boundary plays an important role in text-to-speech synthesis (TTS) in terms of naturalness and readability. However, the acquisition of prosodic boundary labels relies on manual annotation, which is costly and time-consuming. In this paper, we propose to automatically extract prosodic boundary labels from text-audio data via a neural text-speech model with pre-trained audio encoders. This model is pre-trained on text and speech data separately and jointly fine-tuned on TTS data in a triplet format: {speech, text, prosody}. The experimental results on both automatic evaluation and human evaluation demonstrate that: 1) the proposed text-speech prosody annotation framework significantly outperforms text-only baselines; 2) the quality of automatic prosodic boundary annotations is comparable to human annotations; 3) TTS systems trained with model-annotated boundaries are slightly better than systems that use manual ones.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 11 of its 27 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

11 of 27 references.

References · 11

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  6. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
  7. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  8. 2020Expressive TTS Training with Frame and Style Reconstruction Losscited by 30
  9. 2020Unified Mandarin TTS Front-end Based on Distilled BERT Modelcited by 18
  10. 2019Predicting Prosodic Prominence from Text with Pre-trained Contextualized Word Representationscited by 14
  11. 2022Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Modelcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 2; 2026: 1.

4 papers.

  1. 2023Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of SSWPcited by 1
  2. 2025Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotationcited by 1
  3. 2025The Impact of Prosodic Segmentation on Speech Synthesis of Spontaneous Speechcited by 0
  4. 2026X2Streaming-TTS: Causal Token-Level Text-to-Speech from Streaming Text with Speech-State Inheritancecited by 0