mlsanityData to 7 Oct 2026

Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of SSWP

arXiv 2309.05423 ↗First posted 11 Sep 2023eess.AS cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

In expressive and controllable Text-to-Speech (TTS), explicit prosodic features significantly improve the naturalness and controllability of synthesised speech. However, manual prosody annotation is labor-intensive and inconsistent. To address this issue, a two-stage automatic annotation pipeline is novelly proposed in this paper. In the first stage, we use contrastive pretraining of Speech-Silence and Word-Punctuation (SSWP) pairs to enhance prosodic information in latent representations. In the second stage, we build a multi-modal prosody annotator, comprising pretrained encoders, a text-speech fusing scheme, and a sequence classifier. Experiments on English prosodic boundaries demonstrate that our method achieves state-of-the-art (SOTA) performance with 0.72 and 0.93 f1 score for Prosodic Word and Prosodic Phrase boundary respectively, while bearing remarkable robustness to data scarcity.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

23 of 38 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  8. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  9. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  10. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  11. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  12. 2021PortaSpeech: Portable and High-Quality Generative Text-to-Speechcited by 47
11 more
  1. 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speechcited by 22
  2. 2023Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictionscited by 22
  3. 2021DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 2021cited by 19
  4. 2023CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-trainingcited by 17
  5. 2022Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speechcited by 13
  6. 2020PPG-based singing voice conversion with adversarial representation learningcited by 12
  7. 2022BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language modelcited by 10
  8. 2022A Character-level Span-based Model for Mandarin Prosodic Structure Predictioncited by 4
  9. 2022Automatic Prosody Annotation with Pre-Trained Text-Speech Modelcited by 4
  10. 2023On granularity of prosodic representations in expressive text-to-speechcited by 2
  11. 2022Low-Resource Mongolian Speech Synthesis Based on Automatic Prosody Annotationcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025The Impact of Prosodic Segmentation on Speech Synthesis of Spontaneous Speechcited by 0