# Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection

arXiv: https://arxiv.org/abs/2003.01309 · HTML: https://mlsanity.com/paper/2003.01309/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Mar 2020.

## Built on by (most cited first, 23 of 23)

- [FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs](https://mlsanity.com/paper/2407.04051/) (2024)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Machine Learning for Stuttering Identification: Review, Challenges and Future Directions](https://mlsanity.com/paper/2107.04057/) (2021)
- [Token-Level Supervised Contrastive Learning for Punctuation Restoration](https://mlsanity.com/paper/2107.09099/) (2021)
- [Capitalization and Punctuation Restoration: a Survey](https://mlsanity.com/paper/2111.10746/) (2021)
- [Automatic punctuation restoration with BERT models](https://mlsanity.com/paper/2101.07343/) (2021)
- [Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection](https://mlsanity.com/paper/2205.00620/) (2022)
- [Large Language Models for Dysfluency Detection in Stuttered Speech](https://mlsanity.com/paper/2406.11025/) (2024)
- [Automatic Speech Recognition with BERT and CTC Transformers: A Review](https://mlsanity.com/paper/2410.09456/) (2024)
- [Discriminative Self-training for Punctuation Prediction](https://mlsanity.com/paper/2104.10339/) (2021)
- [Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejection](https://mlsanity.com/paper/2201.03313/) (2022)
- [Incorporating External POS Tagger for Punctuation Restoration](https://mlsanity.com/paper/2106.06731/) (2021)
- [Four-in-One: A Joint Approach to Inverse Text Normalization, Punctuation, Capitalization, and Disfluency for Automatic Speech Recognition](https://mlsanity.com/paper/2210.15063/) (2022)
- [Advancing Stuttering Detection via Data Augmentation, Class-Balanced Loss and Multi-Contextual Deep Learning](https://mlsanity.com/paper/2302.11343/) (2023)
- [Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization](https://mlsanity.com/paper/2305.12927/) (2023)
- [Full-text Error Correction for Chinese Speech Recognition with Large Language Model](https://mlsanity.com/paper/2409.07790/) (2024)
- [A Context-Aware Feature Fusion Framework for Punctuation Restoration](https://mlsanity.com/paper/2203.12487/) (2022)
- [Mask-combine Decoding and Classification Approach for Punctuation Prediction with real-time Inference Constraints](https://mlsanity.com/paper/2112.08098/) (2021)
- [FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration](https://mlsanity.com/paper/2211.04699/) (2022)
- [MMSD-Net: Towards Multi-modal Stuttering Detection](https://mlsanity.com/paper/2407.11492/) (2024)
- [A light-weight and efficient punctuation and word casing prediction model for on-device streaming ASR](https://mlsanity.com/paper/2407.13142/) (2024)
- [Paediatric-HGNN: A Hybrid Heterogeneous Graph Neural Network for Detecting Disfluency in Children's Speech via Multiscale Acoustic Fusion](https://mlsanity.com/paper/2606.08210/) (2026)

## Abstract (arXiv)

With the increased applications of automatic speech recognition (ASR) in recent years, it is essential to automatically insert punctuation marks and remove disfluencies in transcripts, to improve the readability of the transcripts as well as the performance of subsequent applications, such as machine translation, dialogue systems, and so forth. In this paper, we propose a Controllable Time-delay Transformer (CT-Transformer) model that jointly completes the punctuation prediction and disfluency detection tasks in real time. The CT-Transformer model facilitates freezing partial outputs with controllable time delay to fulfill the real-time constraints in partial decoding required by subsequent applications. We further propose a fast decoding strategy to minimize latency while maintaining competitive performance. Experimental results on the IWSLT2011 benchmark dataset and an in-house Chinese annotated dataset demonstrate that the proposed approach outperforms the previous state-of-the-art models on F-scores and achieves a competitive inference speed.