mlsanityData to 7 Oct 2026

PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS

arXiv 2103.15060 ↗First posted 28 Mar 2021cs.CL cs.SD eess.AScited by 52

Not on the map. Why?

Abstract (arXiv)

This paper introduces PnG BERT, a new encoder model for neural TTS. This model is augmented from the original BERT model, by taking both phoneme and grapheme representations of text as input, as well as the word-level alignment between them. It can be pre-trained on a large text corpus in a self-supervised manner, and fine-tuned in a TTS task. Experimental results show that a neural TTS model using a pre-trained PnG BERT as its encoder yields more natural prosody and more accurate pronunciation than a baseline model using only phoneme input with no pre-training. Subjective side-by-side preference evaluations show that raters have no statistically significant preference between the speech synthesized using a PnG BERT and ground truth recordings from professional speakers.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 26 of its 34 references that are in the map. First posted on arXiv 28 Mar 2021.

Builds on

26 of 34 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  8. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  9. 2018Universal Transformerscited by 613
  10. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  11. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  12. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
14 more
  1. 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 257
  2. 2018Efficient Neural Audio Synthesiscited by 218
  3. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  4. 2017Deep Voice: Real-time Neural Text-to-Speechcited by 83
  5. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  6. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  7. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  8. 2020Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesiscited by 38
  9. 2018Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesiscited by 30
  10. 2020Improving Prosody Modelling with Cross-Utterance BERT Embeddings for End-to-end Speech Synthesiscited by 27
  11. 2019Memory-Efficient Adaptive Optimizationcited by 19
  12. 2020Unified Mandarin TTS Front-end Based on Distilled BERT Modelcited by 18
  13. 2018Representation Mixing for TTS Synthesiscited by 7
  14. 2019Feature reinforcement with word embedding and parsing information in neural TTScited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 18; 2023: 13; 2024: 7; 2025: 4; 2026: 3.

52 papers.

Sort:

By area:

  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  2. 2021A Survey on Neural Speech Synthesiscited by 182
  3. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 151
  4. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  5. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  6. 2021Translatotron 2: High-quality direct speech-to-speech translation with voice preservationcited by 69
  7. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  8. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  9. 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 32
  10. 2023Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictionscited by 22
  11. 2023Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representationscited by 22
  12. 2022Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translationcited by 20
38 more
  1. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
  2. 2023XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speechcited by 16
  3. 2021NWT: Towards natural audio-to-video generation with representation learningcited by 14
  4. 2022Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speechcited by 13
  5. 2024FLEURS-R: A Restored Multilingual Speech Corpus for Generation Taskscited by 13
  6. 2021More than Words: In-the-Wild Visually-Driven Prosody for Text-to-Speechcited by 12
  7. 2022Simple and Effective Unsupervised Speech Synthesiscited by 12
  8. 2023Translatotron 3: Speech to Speech Translation with Monolingual Datacited by 12
  9. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  10. 2022Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptationcited by 10
  11. 2022Speech Aware Dialog System Technology Challenge (DSTC11)cited by 10
  12. 2023Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretrainingcited by 10
  13. 2022Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTScited by 6
  14. 2024Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfercited by 6
  15. 2022Simple and Effective Multi-sentence TTS with Expressive and Coherent Prosodycited by 5
  16. 2025Voice Cloning: Comprehensive Surveycited by 4
  17. 2021Neural Lexicon Reader: Reduce Pronunciation Errors in End-to-end TTS by Leveraging External Textual Knowledgecited by 3
  18. 2022Learning the joint distribution of two sequences using little or no paired datacited by 3
  19. 2024A multilingual training strategy for low resource Text to Speechcited by 3
  20. 2021Improving Prosody for Unseen Texts in Speech Synthesis by Utilizing Linguistic Information and Noisy Datacited by 2
  21. 2022A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesiscited by 2
  22. 2022Investigation of Japanese PnG BERT language model in text-to-speech synthesis for pitch accent languagecited by 2
  23. 2022Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencodercited by 2
  24. 2023ELF: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesiscited by 2
  25. 2024Word Boundary Information Isn't Useful for Encoder Language Modelscited by 2
  26. 2025Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioningcited by 2
  27. 2022Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speechcited by 1
  28. 2023QS-TTS: Towards Semi-Supervised Text-to-Speech Synthesis via Vector-Quantized Self-Supervised Speech Representation Learningcited by 1
  29. 2023Cross-Utterance Conditioned VAE for Speech Generationcited by 1
  30. 2023Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of SSWPcited by 1
  31. 2025Comparative Evaluation of Expressive Japanese Character Text-to-Speech with VITS and Style-BERT-VITS2cited by 1
  32. 2022Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speechcited by 0
  33. 2022RWEN-TTS: Relation-aware Word Encoding Network for Natural Text-to-Speech Synthesiscited by 0
  34. 2024Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representationscited by 0
  35. 2024PRESENT: Zero-Shot Text-to-Prosody Controlcited by 0
  36. 2024Cross-Dialect Text-To-Speech in Pitch-Accent Language Incorporating Multi-Dialect Phoneme-Level BERTcited by 0
  37. 2025Multi-task Pretraining for Enhancing Interpretable L2 Pronunciation Assessmentcited by 0
  38. 2026MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Controlcited by 0

50 most cited of 52 All 52 (JSON)