# Capitalization Normalization for Language Modeling with an Accurate and Efficient Hierarchical RNN Model

arXiv: https://arxiv.org/abs/2202.08171 · HTML: https://mlsanity.com/paper/2202.08171/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 14 of 14)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Exploring the Limits of Language Modeling](https://mlsanity.com/paper/1602.02410/) (2016)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Corpora Generation for Grammatical Error Correction](https://mlsanity.com/paper/1904.05780/) (2019)
- [Federated Learning of N-gram Language Models](https://mlsanity.com/paper/1910.03432/) (2019)
- [Improving Performance of End-to-End ASR on Numeric Sequences](https://mlsanity.com/paper/1907.01372/) (2019)
- [Robust Prediction of Punctuation and Truecasing for Medical ASR](https://mlsanity.com/paper/2007.02025/) (2020)
- [Fast and Accurate Capitalization and Punctuation for Automatic Speech Recognition Using Transformer and Chunk Merging](https://mlsanity.com/paper/1908.02404/) (2019)
- [Mobile Keyboard Input Decoding with Finite-State Transducers](https://mlsanity.com/paper/1704.03987/) (2017)
- [An Efficient Architecture for Predicting the Case of Characters using Sequence Models](https://mlsanity.com/paper/2002.00738/) (2020)

## Built on by (most cited first, 3 of 3)

- [Four-in-One: A Joint Approach to Inverse Text Normalization, Punctuation, Capitalization, and Disfluency for Automatic Speech Recognition](https://mlsanity.com/paper/2210.15063/) (2022)
- [Text Injection for Capitalization and Turn-Taking Prediction in Speech Models](https://mlsanity.com/paper/2308.07395/) (2023)
- [AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach](https://mlsanity.com/paper/2607.28175/) (2026)

## Abstract (arXiv)

Capitalization normalization (truecasing) is the task of restoring the correct case (uppercase or lowercase) of noisy text. We propose a fast, accurate and compact two-level hierarchical word-and-character-based recurrent neural network model. We use the truecaser to normalize user-generated text in a Federated Learning framework for language modeling. A case-aware language model trained on this normalized text achieves the same perplexity as a model trained on text with gold capitalization. In a real user A/B experiment, we demonstrate that the improvement translates to reduced prediction error rates in a virtual keyboard application. Similarly, in an ASR language model fusion experiment, we show reduction in uppercase character error rate and word error rate.