# Improving Punctuation Restoration for Speech Transcripts via External Data

arXiv: https://arxiv.org/abs/2110.00560 · HTML: https://mlsanity.com/paper/2110.00560/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [SummScreen: A Dataset for Abstractive Screenplay Summarization](https://mlsanity.com/paper/2104.07091/) (2021)
- [TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection](https://mlsanity.com/paper/1911.04118/) (2019)
- [WSL-DS: Weakly Supervised Learning with Distant Supervision for Query Focused Multi-Document Abstractive Summarization](https://mlsanity.com/paper/2011.01421/) (2020)
- [Punctuation Prediction Model for Conversational Speech](https://mlsanity.com/paper/1807.00543/) (2018)
- [Using Similarity Measures to Select Pretraining Data for NER](https://mlsanity.com/paper/1904.00585/) (2019)
- [Adversarial Transfer Learning for Punctuation Restoration](https://mlsanity.com/paper/2004.00248/) (2020)
- [Token-Level Supervised Contrastive Learning for Punctuation Restoration](https://mlsanity.com/paper/2107.09099/) (2021)
- [Automatic punctuation restoration with BERT models](https://mlsanity.com/paper/2101.07343/) (2021)
- [Discriminative Self-training for Punctuation Prediction](https://mlsanity.com/paper/2104.10339/) (2021)

## Built on by (most cited first, 8 of 8)

- [Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization](https://mlsanity.com/paper/2112.11670/) (2021)
- [An Effective, Performant Named Entity Recognition System for Noisy Business Telephone Conversation Transcripts](https://mlsanity.com/paper/2209.13736/) (2022)
- [Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loop](https://mlsanity.com/paper/2211.01354/) (2022)
- [Entity-level Sentiment Analysis in Contact Center Telephone Conversations](https://mlsanity.com/paper/2210.13401/) (2022)
- [Resolving Transcription Ambiguity in Spanish: A Hybrid Acoustic-Lexical System for Punctuation Restoration](https://mlsanity.com/paper/2402.03519/) (2024)
- [Punctuation Restoration in Spanish Customer Support Transcripts using Transfer Learning](https://mlsanity.com/paper/2205.13961/) (2022)
- [AgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach](https://mlsanity.com/paper/2607.28175/) (2026)
- [A Small and Fast BERT for Chinese Medical Punctuation Restoration](https://mlsanity.com/paper/2308.12568/) (2023)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) systems generally do not produce punctuated transcripts. To make transcripts more readable and follow the expected input format for downstream language models, it is necessary to add punctuation marks. In this paper, we tackle the punctuation restoration problem specifically for the noisy text (e.g., phone conversation scenarios). To leverage the available written text datasets, we introduce a data sampling technique based on an n-gram language model to sample more training data that are similar to our in-domain data. Moreover, we propose a two-stage fine-tuning approach that utilizes the sampled external data as well as our in-domain dataset for models based on BERT. Extensive experiments show that the proposed approach outperforms the baseline with an improvement of 1:12% F1 score.