# Neural Grapheme-to-Phoneme Conversion with Pre-trained Grapheme Models

arXiv: https://arxiv.org/abs/2201.10716 · HTML: https://mlsanity.com/paper/2201.10716/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- [Transformer based Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/2004.06338/) (2020)
- [Massively Multilingual Neural Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/1708.01464/) (2017)
- [Jointly Learning to Align and Convert Graphemes to Phonemes with Neural Attention Models](https://mlsanity.com/paper/1610.06540/) (2016)
- [Grapheme-to-Phoneme Transformer Model for Transfer Learning Dialects](https://mlsanity.com/paper/2104.04091/) (2021)

## Built on by (most cited first, 4 of 4)

- [Improving grapheme-to-phoneme conversion by learning pronunciations from speech recordings](https://mlsanity.com/paper/2307.16643/) (2023)
- [Multilingual context-based pronunciation learning for Text-to-Speech](https://mlsanity.com/paper/2307.16709/) (2023)
- [grapheme-kit: Grapheme-Level Metrics and Text Processing for Multilingual NLP](https://mlsanity.com/paper/2607.22456/) (2026)
- [Phonemizing User-Generated Text: A Benchmark, Taxonomy, and Compositional Approach](https://mlsanity.com/paper/2609.27205/) (2026)

## Abstract (arXiv)

Neural network models have achieved state-of-the-art performance on grapheme-to-phoneme (G2P) conversion. However, their performance relies on large-scale pronunciation dictionaries, which may not be available for a lot of languages. Inspired by the success of the pre-trained language model BERT, this paper proposes a pre-trained grapheme model called grapheme BERT (GBERT), which is built by self-supervised training on a large, language-specific word list with only grapheme information. Furthermore, two approaches are developed to incorporate GBERT into the state-of-the-art Transformer-based G2P model, i.e., fine-tuning GBERT or fusing GBERT into the Transformer model by attention. Experimental results on the Dutch, Serbo-Croatian, Bulgarian and Korean datasets of the SIGMORPHON 2021 G2P task confirm the effectiveness of our GBERT-based G2P models under both medium-resource and low-resource data conditions.