# Shallow Fusion of Weighted Finite-State Transducer and Language Model for Text Normalization

arXiv: https://arxiv.org/abs/2203.15917 · HTML: https://mlsanity.com/paper/2203.15917/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [NeMo Inverse Text Normalization: From Development To Production](https://mlsanity.com/paper/2104.05055/) (2021)
- [Neural Inverse Text Normalization](https://mlsanity.com/paper/2102.06380/) (2021)
- [A Unified Transformer-based Framework for Duplex Text Normalization](https://mlsanity.com/paper/2108.09889/) (2021)

## Built on by (most cited first, 8 of 8)

- [DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors](https://mlsanity.com/paper/2406.11427/) (2024)
- [USAT: A Universal Speaker-Adaptive Text-to-Speech Approach](https://mlsanity.com/paper/2404.18094/) (2024)
- [The ParlaSpeech Collection of Automatically Generated Speech and Text Datasets from Parliamentary Proceedings](https://mlsanity.com/paper/2409.15397/) (2024)
- [SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation](https://mlsanity.com/paper/2403.08196/) (2024)
- [Non-autoregressive real-time Accent Conversion model with voice cloning](https://mlsanity.com/paper/2405.13162/) (2024)
- [SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue](https://mlsanity.com/paper/2603.16783/) (2026)
- [Complex-Text Robustness Evaluation and Failure Diagnosis for Low-Resource Multilingual Text-to-Speech](https://mlsanity.com/paper/2609.11545/) (2026)
- [THA: Weighted Finite-State Text Normalization and Inverse Text Normalization for Khmer](https://mlsanity.com/paper/2609.30984/) (2026)

## Abstract (arXiv)

Text normalization (TN) systems in production are largely rule-based using weighted finite-state transducers (WFST). However, WFST-based systems struggle with ambiguous input when the normalized form is context-dependent. On the other hand, neural text normalization systems can take context into account but they suffer from unrecoverable errors and require labeled normalization datasets, which are hard to collect. We propose a new hybrid approach that combines the benefits of rule-based and neural systems. First, a non-deterministic WFST outputs all normalization candidates, and then a neural language model picks the best one -- similar to shallow fusion for automatic speech recognition. While the WFST prevents unrecoverable errors, the language model resolves contextual ambiguity. The approach is easy to extend and we show it is effective. It achieves comparable or better results than existing state-of-the-art TN models.