mlsanityData to 7 Oct 2026

Token-Level Supervised Contrastive Learning for Punctuation Restoration

arXiv 2107.09099 ↗First posted 19 Jul 2021cs.CL cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Punctuation is critical in understanding natural language text. Currently, most automatic speech recognition (ASR) systems do not generate punctuation, which affects the performance of downstream tasks, such as intent detection and slot filling. This gives rise to the need for punctuation restoration. Recent work in punctuation restoration heavily utilizes pre-trained language models without considering data imbalance when predicting punctuation classes. In this work, we address this problem by proposing a token-level supervised contrastive learning method that aims at maximizing the distance of representation of different punctuation marks in the embedding space. The result shows that training with token-level supervised contrastive learning obtains up to 3.2% absolute F1 improvement on the test set.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 13 of its 33 references that are in the map. First posted on arXiv 19 Jul 2021.

Builds on

13 of 33 references.

References · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  6. 2020Supervised Contrastive Learningcited by 2,803
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  9. 2020Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detectioncited by 23
  10. 2018Punctuation Prediction Model for Conversational Speechcited by 20
  11. 2016TristouNet: Triplet Loss for Speaker Turn Embeddingcited by 18
  12. 2020Adversarial Transfer Learning for Punctuation Restorationcited by 15
1 more
  1. 2021Automatic punctuation restoration with BERT modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 4; 2023: 3; 2024: 2; 2025: 0; 2026: 1.

14 papers.

Sort:

By area:

  1. 2021BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamesecited by 33
  2. 2022Superficial White Matter Analysis: An Efficient Point-cloud-based Deep Learning Framework with Supervised Contrastive Learning for Consistent Tractography Parcellation across Populations and dMRI Acquisitionscited by 19
  3. 2022Whodunit? Learning to Contrast for Authorship Attributioncited by 17
  4. 2021CL4AC: A Contrastive Loss for Audio Captioningcited by 16
  5. 2021Improving Punctuation Restoration for Speech Transcripts via External Datacited by 8
  6. 2022A Context-Aware Feature Fusion Framework for Punctuation Restorationcited by 2
  7. 2023Efficient Ensemble for Multimodal Punctuation Restoration using Time-Delay Neural Networkcited by 1
  8. 2023End-to-end Joint Punctuated and Normalized ASR with a Limited Amount of Punctuated Training Datacited by 1
  9. 2021Supervised Graph Contrastive Pretraining for Text Classificationcited by 0
  10. 2022FF2: A Feature Fusion Two-Stream Framework for Punctuation Restorationcited by 0
  11. 2023A Small and Fast BERT for Chinese Medical Punctuation Restorationcited by 0
  12. 2024A light-weight and efficient punctuation and word casing prediction model for on-device streaming ASRcited by 0
2 more
  1. 2024Spontaneous Informal Speech Dataset for Punctuation Restorationcited by 0
  2. 2026SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularizationcited by 0