# Token-Level Supervised Contrastive Learning for Punctuation Restoration

arXiv: https://arxiv.org/abs/2107.09099 · HTML: https://mlsanity.com/paper/2107.09099/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection](https://mlsanity.com/paper/2003.01309/) (2020)
- [Punctuation Prediction Model for Conversational Speech](https://mlsanity.com/paper/1807.00543/) (2018)
- [TristouNet: Triplet Loss for Speaker Turn Embedding](https://mlsanity.com/paper/1609.04301/) (2016)
- [Adversarial Transfer Learning for Punctuation Restoration](https://mlsanity.com/paper/2004.00248/) (2020)
- [Automatic punctuation restoration with BERT models](https://mlsanity.com/paper/2101.07343/) (2021)

## Built on by (most cited first, 14 of 14)

- [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://mlsanity.com/paper/2109.09701/) (2021)
- [Superficial White Matter Analysis: An Efficient Point-cloud-based Deep Learning Framework with Supervised Contrastive Learning for Consistent Tractography Parcellation across Populations and dMRI Acquisitions](https://mlsanity.com/paper/2207.08975/) (2022)
- [Whodunit? Learning to Contrast for Authorship Attribution](https://mlsanity.com/paper/2209.11887/) (2022)
- [CL4AC: A Contrastive Loss for Audio Captioning](https://mlsanity.com/paper/2107.09990/) (2021)
- [Improving Punctuation Restoration for Speech Transcripts via External Data](https://mlsanity.com/paper/2110.00560/) (2021)
- [A Context-Aware Feature Fusion Framework for Punctuation Restoration](https://mlsanity.com/paper/2203.12487/) (2022)
- [Efficient Ensemble for Multimodal Punctuation Restoration using Time-Delay Neural Network](https://mlsanity.com/paper/2302.13376/) (2023)
- [End-to-end Joint Punctuated and Normalized ASR with a Limited Amount of Punctuated Training Data](https://mlsanity.com/paper/2311.17741/) (2023)
- [Supervised Graph Contrastive Pretraining for Text Classification](https://mlsanity.com/paper/2112.11389/) (2021)
- [FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration](https://mlsanity.com/paper/2211.04699/) (2022)
- [A Small and Fast BERT for Chinese Medical Punctuation Restoration](https://mlsanity.com/paper/2308.12568/) (2023)
- [A light-weight and efficient punctuation and word casing prediction model for on-device streaming ASR](https://mlsanity.com/paper/2407.13142/) (2024)
- [Spontaneous Informal Speech Dataset for Punctuation Restoration](https://mlsanity.com/paper/2409.11241/) (2024)
- [SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularization](https://mlsanity.com/paper/2605.08809/) (2026)

## Abstract (arXiv)

Punctuation is critical in understanding natural language text. Currently, most automatic speech recognition (ASR) systems do not generate punctuation, which affects the performance of downstream tasks, such as intent detection and slot filling. This gives rise to the need for punctuation restoration. Recent work in punctuation restoration heavily utilizes pre-trained language models without considering data imbalance when predicting punctuation classes. In this work, we address this problem by proposing a token-level supervised contrastive learning method that aims at maximizing the distance of representation of different punctuation marks in the embedding space. The result shows that training with token-level supervised contrastive learning obtains up to 3.2% absolute F1 improvement on the test set.