# Towards Efficient Simultaneous Inverse Text Normalization with Pretrained Text-to-Text Language Model and Read-Tag-Write Policy

arXiv: https://arxiv.org/abs/2610.19492 · HTML: https://mlsanity.com/paper/2610.19492/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/1906.05218/) (2019)
- [Efficient Wait-k Models for Simultaneous Machine Translation](https://mlsanity.com/paper/2005.08595/) (2020)
- [Simpler and Faster Learning of Adaptive Policies for Simultaneous Translation](https://mlsanity.com/paper/1909.01559/) (2019)
- [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://mlsanity.com/paper/2109.09701/) (2021)
- [NeMo Inverse Text Normalization: From Development To Production](https://mlsanity.com/paper/2104.05055/) (2021)
- [Hidden Markov Transformer for Simultaneous Machine Translation](https://mlsanity.com/paper/2303.00257/) (2023)
- [Neural Inverse Text Normalization](https://mlsanity.com/paper/2102.06380/) (2021)
- [PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation](https://mlsanity.com/paper/2110.12199/) (2021)
- [Improving Vietnamese Named Entity Recognition from Speech Using Word Capitalization and Punctuation Recovery Models](https://mlsanity.com/paper/2010.00198/) (2020)
- [Adaptive Policy with Wait-k Model for Simultaneous Translation](https://mlsanity.com/paper/2310.14853/) (2023)
- [Thutmose Tagger: Single-pass neural model for Inverse Text Normalization](https://mlsanity.com/paper/2208.00064/) (2022)
- [MTet: Multi-domain Translation for English and Vietnamese](https://mlsanity.com/paper/2210.05610/) (2022)
- [Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation](https://mlsanity.com/paper/2501.00868/) (2025)
- [Streaming, fast and accurate on-device Inverse Text Normalization for Automatic Speech Recognition](https://mlsanity.com/paper/2211.03721/) (2022)
- [ChunkFormer: Masked Chunking Conformer For Long-Form Speech Transcription](https://mlsanity.com/paper/2502.14673/) (2025)
- [Improving Streaming Speech Recognition With Time-Shifted Contextual Attention And Dynamic Right Context Masking](https://mlsanity.com/paper/2502.15158/) (2025)
- [Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization](https://mlsanity.com/paper/2505.24229/) (2025)
- [Language Agnostic Data-Driven Inverse Text Normalization](https://mlsanity.com/paper/2301.08506/) (2023)

## Abstract (arXiv)

Streaming Inverse Text Normalization (ITN) is vital for converting spoken-form outputs from streaming Automatic Speech Recognition into formatted written text. Existing streaming ITN methods rely on hybrid systems combining neural tagging with expert-crafted finite-state transducer rules, limiting scalability across domains and languages. While end-to-end models offer superior scalability by learning directly from data, standard encoder-decoder architectures are inherently non-streaming due to global attention. In this paper, we propose an efficient streaming end-to-end ITN system, adapting a pretrained text-to-text model to leverage its robust linguistic knowledge. To enable streaming, we introduce architectural adaptations, a specialized training strategy, a Read-Tag-Write decoding policy, and inference optimizations. Experiments on Vietnamese datasets show accuracy comparable to non-streaming baselines, outperforming hybrid methods while satisfying real-time latency requirements.