# Incorporating External POS Tagger for Punctuation Restoration

arXiv: https://arxiv.org/abs/2106.06731 · HTML: https://mlsanity.com/paper/2106.06731/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 12 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://mlsanity.com/paper/2006.03236/) (2020)
- [Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection](https://mlsanity.com/paper/2003.01309/) (2020)
- [Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech](https://mlsanity.com/paper/2008.00702/) (2020)
- [Adversarial Transfer Learning for Punctuation Restoration](https://mlsanity.com/paper/2004.00248/) (2020)
- [Joint Learning of Correlated Sequence Labelling Tasks Using Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1703.04650/) (2017)

## Built on by (most cited first, 5 of 5)

- [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://mlsanity.com/paper/2109.09701/) (2021)
- [A Context-Aware Feature Fusion Framework for Punctuation Restoration](https://mlsanity.com/paper/2203.12487/) (2022)
- [Text Editing as Imitation Game](https://mlsanity.com/paper/2210.12276/) (2022)
- [FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration](https://mlsanity.com/paper/2211.04699/) (2022)
- [A Small and Fast BERT for Chinese Medical Punctuation Restoration](https://mlsanity.com/paper/2308.12568/) (2023)

## Abstract (arXiv)

Punctuation restoration is an important post-processing step in automatic speech recognition. Among other kinds of external information, part-of-speech (POS) taggers provide informative tags, suggesting each input token's syntactic role, which has been shown to be beneficial for the punctuation restoration task. In this work, we incorporate an external POS tagger and fuse its predicted labels into the existing language model to provide syntactic information. Besides, we propose sequence boundary sampling (SBS) to learn punctuation positions more efficiently as a sequence tagging task. Experimental results show that our methods can consistently obtain performance gains and achieve a new state-of-the-art on the common IWSLT benchmark. Further ablation studies illustrate that both large pre-trained language models and the external POS tagger take essential parts to improve the model's performance.