# Streaming Punctuation: A Novel Punctuation Technique Leveraging Bidirectional Context for Continuous Speech Recognition

arXiv: https://arxiv.org/abs/2301.03819 · HTML: https://mlsanity.com/paper/2301.03819/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 33 references that are in the map.
- First posted on arXiv 10 Jan 2023.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Punctuation Prediction Model for Conversational Speech](https://mlsanity.com/paper/1807.00543/) (2018)
- [E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASR](https://mlsanity.com/paper/2204.10749/) (2022)
- [Capitalization and Punctuation Restoration: a Survey](https://mlsanity.com/paper/2111.10746/) (2021)
- [Automatic punctuation restoration with BERT models](https://mlsanity.com/paper/2101.07343/) (2021)
- [Joint prediction of truecasing and punctuation for conversational speech in low-resource scenarios](https://mlsanity.com/paper/2109.06103/) (2021)
- [Streaming Punctuation for Long-form Dictation with Transformers](https://mlsanity.com/paper/2210.05756/) (2022)
- [Smart Speech Segmentation using Acousto-Linguistic Features with look-ahead](https://mlsanity.com/paper/2210.14446/) (2022)
- [A Context-Aware Feature Fusion Framework for Punctuation Restoration](https://mlsanity.com/paper/2203.12487/) (2022)

## Built on by (most cited first, 2 of 2)

- [Semantic Segmentation with Bidirectional Language Models Improves Long-form ASR](https://mlsanity.com/paper/2305.18419/) (2023)
- [Non-verbal information in spontaneous speech -- towards a new framework of analysis](https://mlsanity.com/paper/2403.03522/) (2024)

## Abstract (arXiv)

While speech recognition Word Error Rate (WER) has reached human parity for English, continuous speech recognition scenarios such as voice typing and meeting transcriptions still suffer from segmentation and punctuation problems, resulting from irregular pausing patterns or slow speakers. Transformer sequence tagging models are effective at capturing long bi-directional context, which is crucial for automatic punctuation. Automatic Speech Recognition (ASR) production systems, however, are constrained by real-time requirements, making it hard to incorporate the right context when making punctuation decisions. Context within the segments produced by ASR decoders can be helpful but limiting in overall punctuation performance for a continuous speech session. In this paper, we propose a streaming approach for punctuation or re-punctuation of ASR output using dynamic decoding windows and measure its impact on punctuation and segmentation accuracy across scenarios. The new system tackles over-segmentation issues, improving segmentation F0.5-score by 13.9%. Streaming punctuation achieves an average BLEUscore improvement of 0.66 for the downstream task of Machine Translation (MT).