# Unified Mandarin TTS Front-end Based on Distilled BERT Model

arXiv: https://arxiv.org/abs/2012.15404 · HTML: https://mlsanity.com/paper/2012.15404/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Dec 2020.

## Built on by (most cited first, 18 of 18)

- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS](https://mlsanity.com/paper/2103.15060/) (2021)
- [Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions](https://mlsanity.com/paper/2301.08810/) (2023)
- [Review of end-to-end speech synthesis technology based on deep learning](https://mlsanity.com/paper/2104.09995/) (2021)
- [Text-driven Emotional Style Control and Cross-speaker Style Transfer in Neural TTS](https://mlsanity.com/paper/2207.06000/) (2022)
- [A Character-level Span-based Model for Mandarin Prosodic Structure Prediction](https://mlsanity.com/paper/2203.16922/) (2022)
- [Automatic Prosody Annotation with Pre-Trained Text-Speech Model](https://mlsanity.com/paper/2206.07956/) (2022)
- [A unified front-end framework for English text-to-speech synthesis](https://mlsanity.com/paper/2305.10666/) (2023)
- [Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Models](https://mlsanity.com/paper/2506.00832/) (2025)
- [Improving Prosody for Unseen Texts in Speech Synthesis by Utilizing Linguistic Information and Noisy Data](https://mlsanity.com/paper/2111.07549/) (2021)
- [Cross-Lingual Transfer Learning for Phrase Break Prediction with Multilingual Language Model](https://mlsanity.com/paper/2306.02579/) (2023)
- [Multilingual context-based pronunciation learning for Text-to-Speech](https://mlsanity.com/paper/2307.16709/) (2023)
- [A Polyphone BERT for Polyphone Disambiguation in Mandarin Chinese](https://mlsanity.com/paper/2207.12089/) (2022)
- [Efficient Incremental Text-to-Speech on GPUs](https://mlsanity.com/paper/2211.13939/) (2022)
- [Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information](https://mlsanity.com/paper/2308.16577/) (2023)
- [Cross-Utterance Conditioned VAE for Speech Generation](https://mlsanity.com/paper/2309.04156/) (2023)
- [Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotation](https://mlsanity.com/paper/2506.07646/) (2025)
- [Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech](https://mlsanity.com/paper/2206.02147/) (2022)

## Abstract (arXiv)

The front-end module in a typical Mandarin text-to-speech system (TTS) is composed of a long pipeline of text processing components, which requires extensive efforts to build and is prone to large accumulative model size and cascade errors. In this paper, a pre-trained language model (PLM) based model is proposed to simultaneously tackle the two most important tasks in TTS front-end, i.e., prosodic structure prediction (PSP) and grapheme-to-phoneme (G2P) conversion. We use a pre-trained Chinese BERT[1] as the text encoder and employ multi-task learning technique to adapt it to the two TTS front-end tasks. Then, the BERT encoder is distilled into a smaller model by employing a knowledge distillation technique called TinyBERT[2], making the whole model size 25% of that of benchmark pipeline models while maintaining competitive performance on both tasks. With the proposed the methods, we are able to run the whole TTS front-end module in a light and unified manner, which is more friendly to deployment on mobile devices.