# Augmenting Automatic Speech Recognition Models with Disfluency Detection

arXiv: https://arxiv.org/abs/2409.10177 · HTML: https://mlsanity.com/paper/2409.10177/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Builds on (most cited first, 16 of 16)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://mlsanity.com/paper/2007.09127/) (2020)
- [Flashlight: Enabling Innovation in Tools for Machine Learning](https://mlsanity.com/paper/2201.12465/) (2022)
- [Unconstrained Dysfluency Modeling for Dysfluent Speech Transcription and Detection](https://mlsanity.com/paper/2312.12810/) (2023)
- [From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition](https://mlsanity.com/paper/2302.09044/) (2023)
- [Disfluency Detection with Unlabeled Data and Small BERT Models](https://mlsanity.com/paper/2104.10769/) (2021)
- [Towards Hierarchical Spoken Language Dysfluency Modeling](https://mlsanity.com/paper/2401.10015/) (2024)
- [Adapting an ASR Foundation Model for Spoken Language Assessment](https://mlsanity.com/paper/2307.09378/) (2023)
- [Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection](https://mlsanity.com/paper/2205.00620/) (2022)
- [Enhancing ASR for Stuttered Speech with Limited Data Using Detect and Pass](https://mlsanity.com/paper/2202.05396/) (2022)
- [Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling](https://mlsanity.com/paper/2306.00996/) (2023)
- [Studying Alignment in a Collaborative Learning Activity via Automatic Methods: The Link Between What We Say and Do](https://mlsanity.com/paper/2104.04429/) (2021)
- [Streaming Joint Speech Recognition and Disfluency Detection](https://mlsanity.com/paper/2211.08726/) (2022)

## Built on by (most cited first, 6 of 6)

- [Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing](https://mlsanity.com/paper/2607.18934/) (2026)
- [Automatic Speech Recognition for Non-Native English: Accuracy and Disfluency Handling](https://mlsanity.com/paper/2503.06924/) (2025)
- [On the Difficulty of Token-Level Modeling of Dysfluency and Fluency Shaping Artifacts](https://mlsanity.com/paper/2512.02027/) (2025)
- [Stuttering-Aware Automatic Speech Recognition for Indonesian Language](https://mlsanity.com/paper/2601.03727/) (2026)
- [The Role of Disfluencies in Speech Translation](https://mlsanity.com/paper/2608.02138/) (2026)
- [Beyond WER: Entity and Disfluency Recall in Accented Conversational ASR](https://mlsanity.com/paper/2609.20828/) (2026)

## Abstract (arXiv)

Speech disfluency commonly occurs in conversational and spontaneous speech. However, standard Automatic Speech Recognition (ASR) models struggle to accurately recognize these disfluencies because they are typically trained on fluent transcripts. Current research mainly focuses on detecting disfluencies within transcripts, overlooking their exact location and duration in the speech. Additionally, previous work often requires model fine-tuning and addresses limited types of disfluencies. In this work, we present an inference-only approach to augment any ASR model with the ability to detect open-set disfluencies. We first demonstrate that ASR models have difficulty transcribing speech disfluencies. Next, this work proposes a modified Connectionist Temporal Classification(CTC)-based forced alignment algorithm from citekurzinger2020ctc to predict word-level timestamps while effectively capturing disfluent speech. Additionally, we develop a model to classify alignment gaps between timestamps as either containing disfluent speech or silence. This model achieves an accuracy of 81.62% and an F1-score of 80.07%. We test the augmentation pipeline of alignment gap detection and classification on a disfluent dataset. Our results show that we captured 74.13% of the words that were initially missed by the transcription, demonstrating the potential of this pipeline for downstream tasks.