# Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization

arXiv: https://arxiv.org/abs/2505.24229 · HTML: https://mlsanity.com/paper/2505.24229/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PhoBERT: Pre-trained language models for Vietnamese](https://mlsanity.com/paper/2003.00744/) (2020)
- [NeMo Inverse Text Normalization: From Development To Production](https://mlsanity.com/paper/2104.05055/) (2021)
- [Neural Inverse Text Normalization](https://mlsanity.com/paper/2102.06380/) (2021)
- [Variable Attention Masking for Configurable Transformer Transducer Speech Recognition](https://mlsanity.com/paper/2211.01438/) (2022)
- [Thutmose Tagger: Single-pass neural model for Inverse Text Normalization](https://mlsanity.com/paper/2208.00064/) (2022)
- [Four-in-One: A Joint Approach to Inverse Text Normalization, Punctuation, Capitalization, and Disfluency for Automatic Speech Recognition](https://mlsanity.com/paper/2210.15063/) (2022)
- [Streaming, fast and accurate on-device Inverse Text Normalization for Automatic Speech Recognition](https://mlsanity.com/paper/2211.03721/) (2022)
- [ChunkFormer: Masked Chunking Conformer For Long-Form Speech Transcription](https://mlsanity.com/paper/2502.14673/) (2025)
- [Improving Data Driven Inverse Text Normalization using Data Augmentation](https://mlsanity.com/paper/2207.09674/) (2022)
- [Improving Streaming Speech Recognition With Time-Shifted Contextual Attention And Dynamic Right Context Masking](https://mlsanity.com/paper/2502.15158/) (2025)

## Built on by (most cited first, 2 of 2)

- [Dual-Form ASR: Semantics-Aware Inverse Text Normalization for Chinese Speech Recognition](https://mlsanity.com/paper/2609.02901/) (2026)
- [Towards Efficient Simultaneous Inverse Text Normalization with Pretrained Text-to-Text Language Model and Read-Tag-Write Policy](https://mlsanity.com/paper/2610.19492/) (2026)

## Abstract (arXiv)

Inverse Text Normalization (ITN) is crucial for converting spoken Automatic Speech Recognition (ASR) outputs into well-formatted written text, enhancing both readability and usability. Despite its importance, the integration of streaming ITN within streaming ASR remains largely unexplored due to challenges in accuracy, efficiency, and adaptability, particularly in low-resource and limited-context scenarios. In this paper, we introduce a streaming pretrained language model for ITN, leveraging pretrained linguistic representations for improved robustness. To address streaming constraints, we propose Dynamic Context-Aware during training and inference, enabling adaptive chunk size adjustments and the integration of right-context information. Experimental results demonstrate that our method achieves accuracy comparable to non-streaming ITN and surpasses existing streaming ITN models on a Vietnamese dataset, all while maintaining low latency, ensuring seamless integration into ASR systems.