# Automatic Disfluency Detection from Untranscribed Speech

arXiv: https://arxiv.org/abs/2311.00867 · HTML: https://mlsanity.com/paper/2311.00867/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [The Multilingual TEDx Corpus for Speech Recognition and Translation](https://mlsanity.com/paper/2102.01757/) (2021)
- [SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network](https://mlsanity.com/paper/2104.02133/) (2021)
- [TorchAudio: Building Blocks for Audio and Speech Processing](https://mlsanity.com/paper/2110.15018/) (2021)
- [SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stutter](https://mlsanity.com/paper/2102.12394/) (2021)
- [Small and Practical BERT Models for Sequence Labeling](https://mlsanity.com/paper/1909.00100/) (2019)
- [Disfluency Detection using a Bidirectional LSTM](https://mlsanity.com/paper/1604.03209/) (2016)
- [Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0](https://mlsanity.com/paper/2204.03417/) (2022)
- [Twitter Bot Detection Using Bidirectional Long Short-term Memory Neural Networks and Word Embeddings](https://mlsanity.com/paper/2002.01336/) (2020)
- [Disfluencies and Human Speech Transcription Errors](https://mlsanity.com/paper/1904.04398/) (2019)
- [Multi-Task Self-Supervised Learning for Disfluency Detection](https://mlsanity.com/paper/1908.05378/) (2019)
- [From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition](https://mlsanity.com/paper/2302.09044/) (2023)
- [Disfluency Detection with Unlabeled Data and Small BERT Models](https://mlsanity.com/paper/2104.10769/) (2021)
- [Disfluency Detection using Auto-Correlational Neural Networks](https://mlsanity.com/paper/1808.09092/) (2018)
- [Analysis and Tuning of a Voice Assistant System for Dysfluent Speech](https://mlsanity.com/paper/2106.11759/) (2021)
- [Giving Attention to the Unexpected: Using Prosody Innovations in Disfluency Detection](https://mlsanity.com/paper/1904.04388/) (2019)
- [Improved Robustness to Disfluencies in RNN-Transducer Based Speech Recognition](https://mlsanity.com/paper/2012.06259/) (2020)
- [Identification of primary and collateral tracks in stuttered speech](https://mlsanity.com/paper/2003.01018/) (2020)

## Built on by (most cited first, 6 of 6)

- [CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions](https://mlsanity.com/paper/2408.16589/) (2024)
- [Whisper Has an Internal Word Aligner](https://mlsanity.com/paper/2509.09987/) (2025)
- [Missingness-resilient Video-enhanced Multimodal Disfluency Detection](https://mlsanity.com/paper/2406.06964/) (2024)
- [Switchboard-Affect: Emotion Perception Labels from Conversational Speech](https://mlsanity.com/paper/2510.13906/) (2025)
- [Umm... With Transformers? Insights from Filled Pause Use across Four Slavic Parliaments](https://mlsanity.com/paper/2607.05964/) (2026)
- [Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents](https://mlsanity.com/paper/2608.22101/) (2026)

## Abstract (arXiv)

Speech disfluencies, such as filled pauses or repetitions, are disruptions in the typical flow of speech. Stuttering is a speech disorder characterized by a high rate of disfluencies, but all individuals speak with some disfluencies and the rates of disfluencies may by increased by factors such as cognitive load. Clinically, automatic disfluency detection may help in treatment planning for individuals who stutter. Outside of the clinic, automatic disfluency detection may serve as a pre-processing step to improve natural language understanding in downstream applications. With this wide range of applications in mind, we investigate language, acoustic, and multimodal methods for frame-level automatic disfluency detection and categorization. Each of these methods relies on audio as an input. First, we evaluate several automatic speech recognition (ASR) systems in terms of their ability to transcribe disfluencies, measured using disfluency error rates. We then use these ASR transcripts as input to a language-based disfluency detection model. We find that disfluency detection performance is largely limited by the quality of transcripts and alignments. We find that an acoustic-based approach that does not require transcription as an intermediate step outperforms the ASR language approach. Finally, we present multimodal architectures which we find improve disfluency detection performance over the unimodal approaches. Ultimately, this work introduces novel approaches for automatic frame-level disfluency and categorization. In the long term, this will help researchers incorporate automatic disfluency detection into a range of applications.