# Discriminative Self-training for Punctuation Prediction

arXiv: https://arxiv.org/abs/2104.10339 · HTML: https://mlsanity.com/paper/2104.10339/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 40 references that are in the map.
- First posted on arXiv 21 Apr 2021.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection](https://mlsanity.com/paper/2003.01309/) (2020)
- [Punctuation Prediction Model for Conversational Speech](https://mlsanity.com/paper/1807.00543/) (2018)
- [Adversarial Transfer Learning for Punctuation Restoration](https://mlsanity.com/paper/2004.00248/) (2020)
- [Fast and Accurate Capitalization and Punctuation for Automatic Speech Recognition Using Transformer and Chunk Merging](https://mlsanity.com/paper/1908.02404/) (2019)
- [Combining Self-Training and Self-Supervised Learning for Unsupervised Disfluency Detection](https://mlsanity.com/paper/2010.15360/) (2020)

## Built on by (most cited first, 6 of 6)

- [Improving Punctuation Restoration for Speech Transcripts via External Data](https://mlsanity.com/paper/2110.00560/) (2021)
- [End-to-end Speech-to-Punctuated-Text Recognition](https://mlsanity.com/paper/2207.03169/) (2022)
- [Automatic Speech Recognition with BERT and CTC Transformers: A Review](https://mlsanity.com/paper/2410.09456/) (2024)
- [Mask-combine Decoding and Classification Approach for Punctuation Prediction with real-time Inference Constraints](https://mlsanity.com/paper/2112.08098/) (2021)
- [FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration](https://mlsanity.com/paper/2211.04699/) (2022)
- [Punctuation Restoration for Singaporean Spoken Languages: English, Malay, and Mandarin](https://mlsanity.com/paper/2212.05356/) (2022)

## Abstract (arXiv)

Punctuation prediction for automatic speech recognition (ASR) output transcripts plays a crucial role for improving the readability of the ASR transcripts and for improving the performance of downstream natural language processing applications. However, achieving good performance on punctuation prediction often requires large amounts of labeled speech transcripts, which is expensive and laborious. In this paper, we propose a Discriminative Self-Training approach with weighted loss and discriminative label smoothing to exploit unlabeled speech transcripts. Experimental results on the English IWSLT2011 benchmark test set and an internal Chinese spoken language dataset demonstrate that the proposed approach achieves significant improvement on punctuation prediction accuracy over strong baselines including BERT, RoBERTa, and ELECTRA models. The proposed Discriminative Self-Training approach outperforms the vanilla self-training approach. We establish a new state-of-the-art (SOTA) on the IWSLT2011 test set, outperforming the current SOTA model by 1.3% absolute gain on F1.