# Transformer-based Models of Text Normalization for Speech Applications

arXiv: https://arxiv.org/abs/2202.00153 · HTML: https://mlsanity.com/paper/2202.00153/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 1 Feb 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning](https://mlsanity.com/paper/1911.03090/) (2019)
- [Seq2Edits: Sequence Transduction Using Span-level Edit Operations](https://mlsanity.com/paper/2009.11136/) (2020)
- [RNN Approaches to Text Normalization: A Challenge](https://mlsanity.com/paper/1611.00068/) (2016)
- [Semi-supervised URL Segmentation with Recurrent Neural Networks Pre-trained on Knowledge Graph Entities](https://mlsanity.com/paper/2011.03138/) (2020)
- [Scalable Multilingual Frontend for TTS](https://mlsanity.com/paper/2004.04934/) (2020)
- [Proteno: Text Normalization with Limited Data for Fast Deployment in Text to Speech Systems](https://mlsanity.com/paper/2104.07777/) (2021)

## Built on by (most cited first, 5 of 5)

- [SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models](https://mlsanity.com/paper/2509.14270/) (2025)
- [A unified front-end framework for English text-to-speech synthesis](https://mlsanity.com/paper/2305.10666/) (2023)
- [Non-Standard Vietnamese Word Detection and Normalization for Text-to-Speech](https://mlsanity.com/paper/2209.02971/) (2022)
- [Multi-Task Learning for Front-End Text Processing in TTS](https://mlsanity.com/paper/2401.06321/) (2024)
- [Positional Description for Numerical Normalization](https://mlsanity.com/paper/2408.12430/) (2024)

## Abstract (arXiv)

Text normalization, or the process of transforming text into a consistent, canonical form, is crucial for speech applications such as text-to-speech synthesis (TTS). In TTS, the system must decide whether to verbalize "1995" as "nineteen ninety five" in "born in 1995" or as "one thousand nine hundred ninety five" in "page 1995". We present an experimental comparison of various Transformer-based sequence-to-sequence (seq2seq) models of text normalization for speech and evaluate them on a variety of datasets of written text aligned to its normalized spoken form. These models include variants of the 2-stage RNN-based tagging/seq2seq architecture introduced by Zhang et al. (2019), where we replace the RNN with a Transformer in one or more stages, as well as vanilla Transformers that output string representations of edit sequences. Of our approaches, using Transformers for sentence context encoding within the 2-stage model proved most effective, with the fine-tuned BERT encoder yielding the best performance.