# Capitalization and Punctuation Restoration: a Survey

arXiv: https://arxiv.org/abs/2111.10746 · HTML: https://mlsanity.com/paper/2111.10746/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 212 references that are in the map.
- First posted on arXiv 21 Nov 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019)
- [Quasi-Recurrent Neural Networks](https://mlsanity.com/paper/1611.01576/) (2016)
- [Tensor2Tensor for Neural Machine Translation](https://mlsanity.com/paper/1803.07416/) (2018)
- [Massive Exploration of Neural Machine Translation Architectures](https://mlsanity.com/paper/1703.03906/) (2017)
- [BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages](https://mlsanity.com/paper/1710.02187/) (2017)
- [Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech](https://mlsanity.com/paper/1803.08976/) (2018)
- [Transition-Based Dependency Parsing with Stack Long Short-Term Memory](https://mlsanity.com/paper/1505.08075/) (2015)
- [Cloze-driven Pretraining of Self-attention Networks](https://mlsanity.com/paper/1903.07785/) (2019)
- [NCRF++: An Open-source Neural Sequence Labeling Toolkit](https://mlsanity.com/paper/1806.05626/) (2018)
- [Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation](https://mlsanity.com/paper/1508.02096/) (2015)
- [Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection](https://mlsanity.com/paper/2003.01309/) (2020)
- [Punctuation Prediction Model for Conversational Speech](https://mlsanity.com/paper/1807.00543/) (2018)
- [Robust Prediction of Punctuation and Truecasing for Medical ASR](https://mlsanity.com/paper/2007.02025/) (2020)
- [ner and pos when nothing is capitalized](https://mlsanity.com/paper/1903.11222/) (2019)

## Built on by (most cited first, 13 of 13)

- [LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models](https://mlsanity.com/paper/2310.02943/) (2023)
- [Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation](https://mlsanity.com/paper/2305.18893/) (2023)
- [Lyrics Transcription for Humans: A Readability-Aware Benchmark](https://mlsanity.com/paper/2408.06370/) (2024)
- [Streaming Punctuation for Long-form Dictation with Transformers](https://mlsanity.com/paper/2210.05756/) (2022)
- [Evaluating OpenAI's Whisper ASR for Punctuation Prediction and Topic Modeling of life histories of the Museum of the Person](https://mlsanity.com/paper/2305.14580/) (2023)
- [Punctuation Restoration Improves Structure Understanding Without Supervision](https://mlsanity.com/paper/2402.08382/) (2024)
- [FullStop:Punctuation and Segmentation Prediction for Dutch with Transformers](https://mlsanity.com/paper/2301.03319/) (2023)
- [Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context for Continuous Speech Recognition](https://mlsanity.com/paper/2301.03819/) (2023)
- [Jam-ALT: A Formatting-Aware Lyrics Transcription Benchmark](https://mlsanity.com/paper/2311.13987/) (2023)
- [Resolving Transcription Ambiguity in Spanish: A Hybrid Acoustic-Lexical System for Punctuation Restoration](https://mlsanity.com/paper/2402.03519/) (2024)
- [Beyond Levenshtein: Leveraging Multiple Algorithms for Robust Word Error Rate Computations And Granular Error Classifications](https://mlsanity.com/paper/2408.15616/) (2024)
- [Enhancing Fully Formatted End-to-End Speech Recognition with Knowledge Distillation via Multi-Codebook Vector Quantization](https://mlsanity.com/paper/2512.18967/) (2025)
- [Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation](https://mlsanity.com/paper/2601.09725/) (2025)

## Abstract (arXiv)

Ensuring proper punctuation and letter casing is a key pre-processing step towards applying complex natural language processing algorithms. This is especially significant for textual sources where punctuation and casing are missing, such as the raw output of automatic speech recognition systems. Additionally, short text messages and micro-blogging platforms offer unreliable and often wrong punctuation and casing. This survey offers an overview of both historical and state-of-the-art techniques for restoring punctuation and correcting word casing. Furthermore, current challenges and research directions are highlighted.