# Automatic punctuation restoration with BERT models

arXiv: https://arxiv.org/abs/2101.07343 · HTML: https://mlsanity.com/paper/2101.07343/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 23 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection](https://mlsanity.com/paper/2003.01309/) (2020)
- [Punctuation Prediction Model for Conversational Speech](https://mlsanity.com/paper/1807.00543/) (2018)
- [Fast and Accurate Capitalization and Punctuation for Automatic Speech Recognition Using Transformer and Chunk Merging](https://mlsanity.com/paper/1908.02404/) (2019)

## Built on by (most cited first, 12 of 12)

- [Token-Level Supervised Contrastive Learning for Punctuation Restoration](https://mlsanity.com/paper/2107.09099/) (2021)
- [Improving Punctuation Restoration for Speech Transcripts via External Data](https://mlsanity.com/paper/2110.00560/) (2021)
- [Streaming Punctuation for Long-form Dictation with Transformers](https://mlsanity.com/paper/2210.05756/) (2022)
- [Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context for Continuous Speech Recognition](https://mlsanity.com/paper/2301.03819/) (2023)
- [ViraPart: A Text Refinement Framework for Automatic Speech Recognition and Natural Language Processing Tasks in Persian](https://mlsanity.com/paper/2110.09086/) (2021)
- [PersianPunc: A Large-Scale Dataset and BERT-Based Approach for Persian Punctuation Restoration](https://mlsanity.com/paper/2603.05314/) (2026)
- [Punctuation restoration in Swedish through fine-tuned KB-BERT](https://mlsanity.com/paper/2202.06769/) (2022)
- [Punctuation Restoration for Singaporean Spoken Languages: English, Malay, and Mandarin](https://mlsanity.com/paper/2212.05356/) (2022)
- [Punctuation restoration Model and Spacing Model for Korean Ancient Document](https://mlsanity.com/paper/2312.11881/) (2023)
- [Beyond the Frame: Single and mutilple video summarization method with user-defined length](https://mlsanity.com/paper/2401.10254/) (2023)
- [A light-weight and efficient punctuation and word casing prediction model for on-device streaming ASR](https://mlsanity.com/paper/2407.13142/) (2024)
- [Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation](https://mlsanity.com/paper/2601.09725/) (2025)

## Abstract (arXiv)

We present an approach for automatic punctuation restoration with BERT models for English and Hungarian. For English, we conduct our experiments on Ted Talks, a commonly used benchmark for punctuation restoration, while for Hungarian we evaluate our models on the Szeged Treebank dataset. Our best models achieve a macro-averaged F1-score of 79.8 in English and 82.2 in Hungarian. Our code is publicly available.