# Neural Inverse Text Normalization arXiv: https://arxiv.org/abs/2102.06380 ยท HTML: https://mlsanity.com/paper/2102.06380/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Builds on 15 of its 22 references that are in the map. - First posted on arXiv 12 Feb 2021. ## Builds on (most cited first, 15 of 15) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017) - [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016) - [A spelling correction model for end-to-end speech recognition](https://mlsanity.com/paper/1902.07178/) (2019) - [Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model](https://mlsanity.com/paper/1910.10697/) (2019) - [RNN Approaches to Text Normalization: A Challenge](https://mlsanity.com/paper/1611.00068/) (2016) - [A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning](https://mlsanity.com/paper/1907.01256/) (2019) - [Improving Readability for Automatic Speech Recognition Transcription](https://mlsanity.com/paper/2004.04438/) (2020) - [Improving Performance of End-to-End ASR on Numeric Sequences](https://mlsanity.com/paper/1907.01372/) (2019) - [Text normalization using memory augmented neural networks](https://mlsanity.com/paper/1806.00044/) (2018) ## Built on by (most cited first, 20 of 20) - [SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition](https://mlsanity.com/paper/2104.02014/) (2021) - [NeMo Inverse Text Normalization: From Development To Production](https://mlsanity.com/paper/2104.05055/) (2021) - [Shallow Fusion of Weighted Finite-State Transducer and Language Model for Text Normalization](https://mlsanity.com/paper/2203.15917/) (2022) - [Unified model for code-switching speech recognition and language identification based on a concatenated tokenizer](https://mlsanity.com/paper/2306.08753/) (2023) - [A Unified Transformer-based Framework for Duplex Text Normalization](https://mlsanity.com/paper/2108.09889/) (2021) - [Thutmose Tagger: Single-pass neural model for Inverse Text Normalization](https://mlsanity.com/paper/2208.00064/) (2022) - [Four-in-One: A Joint Approach to Inverse Text Normalization, Punctuation, Capitalization, and Disfluency for Automatic Speech Recognition](https://mlsanity.com/paper/2210.15063/) (2022) - [Streaming, fast and accurate on-device Inverse Text Normalization for Automatic Speech Recognition](https://mlsanity.com/paper/2211.03721/) (2022) - [Adapting an Unadaptable ASR System](https://mlsanity.com/paper/2306.01208/) (2023) - [Sentence-wise Speech Summarization: Task, Datasets, and End-to-End Modeling with LM Knowledge Distillation](https://mlsanity.com/paper/2408.00205/) (2024) - [AdaCS: Adaptive Normalization for Enhanced Code-Switching ASR](https://mlsanity.com/paper/2501.07102/) (2025) - [Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization](https://mlsanity.com/paper/2505.24229/) (2025) - [Language Agnostic Data-Driven Inverse Text Normalization](https://mlsanity.com/paper/2301.08506/) (2023) - [End-to-end Joint Punctuated and Normalized ASR with a Limited Amount of Punctuated Training Data](https://mlsanity.com/paper/2311.17741/) (2023) - [Zero-Shot Joint Modeling of Multiple Spoken-Text-Style Conversion Tasks using Switching Tokens](https://mlsanity.com/paper/2106.12131/) (2021) - [Improving Robustness of Neural Inverse Text Normalization via Data-Augmentation, Semi-Supervised Learning, and Post-Aligning Method](https://mlsanity.com/paper/2309.08626/) (2023) - [Positional Description for Numerical Normalization](https://mlsanity.com/paper/2408.12430/) (2024) - [Dual-Form ASR: Semantics-Aware Inverse Text Normalization for Chinese Speech Recognition](https://mlsanity.com/paper/2609.02901/) (2026) - [THA: Weighted Finite-State Text Normalization and Inverse Text Normalization for Khmer](https://mlsanity.com/paper/2609.30984/) (2026) - [Towards Efficient Simultaneous Inverse Text Normalization with Pretrained Text-to-Text Language Model and Read-Tag-Write Policy](https://mlsanity.com/paper/2610.19492/) (2026) ## Abstract (arXiv) While there have been several contributions exploring state of the art techniques for text normalization, the problem of inverse text normalization (ITN) remains relatively unexplored. The best known approaches leverage finite state transducer (FST) based models which rely on manually curated rules and are hence not scalable. We propose an efficient and robust neural solution for ITN leveraging transformer based seq2seq models and FST-based text normalization techniques for data preparation. We show that this can be easily extended to other languages without the need for a linguistic expert to manually curate them. We then present a hybrid framework for integrating Neural ITN with an FST to overcome common recoverable errors in production environments. Our empirical evaluations show that the proposed solution minimizes incorrect perturbations (insertions, deletions and substitutions) to ASR output and maintains high quality even on out of domain data. A transformer based model infused with pretraining consistently achieves a lower WER across several datasets and is able to outperform baselines on English, Spanish, German and Italian datasets.