# Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion

arXiv: https://arxiv.org/abs/1904.03446 · HTML: https://mlsanity.com/paper/1904.03446/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2019.

## Built on by (most cited first, 26 of 26)

- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech](https://mlsanity.com/paper/2207.06389/) (2022)
- [Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers](https://mlsanity.com/paper/2405.05945/) (2024)
- [PromptTTS: Controllable Text-to-Speech with Text Descriptions](https://mlsanity.com/paper/2211.12171/) (2022)
- [AdaSpeech: Adaptive Text to Speech for Custom Voice](https://mlsanity.com/paper/2103.00993/) (2021)
- [ByT5 model for massively multilingual grapheme-to-phoneme conversion](https://mlsanity.com/paper/2204.03067/) (2022)
- [Make-A-Voice: Unified Voice Synthesis With Discrete Representation](https://mlsanity.com/paper/2305.19269/) (2023)
- [Training-free Regional Prompting for Diffusion Transformers](https://mlsanity.com/paper/2411.02395/) (2024)
- [RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis](https://mlsanity.com/paper/2404.03204/) (2024)
- [ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer](https://mlsanity.com/paper/2305.12708/) (2023)
- [FlashSpeech: Efficient Zero-Shot Speech Synthesis](https://mlsanity.com/paper/2404.14700/) (2024)
- [ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech](https://mlsanity.com/paper/2202.07816/) (2022)
- [AdaSpeech 2: Adaptive Text to Speech with Untranscribed Data](https://mlsanity.com/paper/2104.09715/) (2021)
- [Denoising with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2410.03755/) (2024)
- [Emotional Prosody Control for Speech Generation](https://mlsanity.com/paper/2111.04730/) (2021)
- [Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation](https://mlsanity.com/paper/2404.14827/) (2024)
- [AdaSpeech 3: Adaptive Text to Speech for Spontaneous Style](https://mlsanity.com/paper/2107.02530/) (2021)
- [Regeneration Learning: A Learning Paradigm for Data Generation](https://mlsanity.com/paper/2301.08846/) (2023)
- [Unified and Effective Ensemble Knowledge Distillation](https://mlsanity.com/paper/2204.00548/) (2022)
- [ParrotTTS: Text-to-Speech synthesis by exploiting self-supervised representations](https://mlsanity.com/paper/2303.01261/) (2023)
- [ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training](https://mlsanity.com/paper/2501.04416/) (2025)
- [Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech](https://mlsanity.com/paper/2206.02147/) (2022)
- [Style Description based Text-to-Speech with Conditional Prosodic Layer Normalization based Diffusion GAN](https://mlsanity.com/paper/2310.18169/) (2023)

## Abstract (arXiv)

Grapheme-to-phoneme (G2P) conversion is an important task in automatic speech recognition and text-to-speech systems. Recently, G2P conversion is viewed as a sequence to sequence task and modeled by RNN or CNN based encoder-decoder framework. However, previous works do not consider the practical issues when deploying G2P model in the production system, such as how to leverage additional unlabeled data to boost the accuracy, as well as reduce model size for online deployment. In this work, we propose token-level ensemble distillation for G2P conversion, which can (1) boost the accuracy by distilling the knowledge from additional unlabeled data, and (2) reduce the model size but maintain the high accuracy, both of which are very practical and helpful in the online production system. We use token-level knowledge distillation, which results in better accuracy than the sequence-level counterpart. What is more, we adopt the Transformer instead of RNN or CNN based models to further boost the accuracy of G2P conversion. Experiments on the publicly available CMUDict dataset and an internal English dataset demonstrate the effectiveness of our proposed method. Particularly, our method achieves 19.88% WER on CMUDict dataset, outperforming the previous works by more than 4.22% WER, and setting the new state-of-the-art results.