# Large Language Models for Dysfluency Detection in Stuttered Speech

arXiv: https://arxiv.org/abs/2406.11025 · HTML: https://mlsanity.com/paper/2406.11025/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://mlsanity.com/paper/2005.10283/) (2020)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [Domain Robustness in Neural Machine Translation](https://mlsanity.com/paper/1911.03109/) (2019)
- [Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation](https://mlsanity.com/paper/2108.04718/) (2021)
- [Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation](https://mlsanity.com/paper/2105.08504/) (2021)
- [SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stutter](https://mlsanity.com/paper/2102.12394/) (2021)
- [Disfluency Detection using a Bidirectional LSTM](https://mlsanity.com/paper/1604.03209/) (2016)
- [Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0](https://mlsanity.com/paper/2204.03417/) (2022)
- [Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection](https://mlsanity.com/paper/2003.01309/) (2020)
- [It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition](https://mlsanity.com/paper/2402.05457/) (2024)
- [KSoF: The Kassel State of Fluency Dataset -- A Therapy Centered Dataset of Stuttering](https://mlsanity.com/paper/2203.05383/) (2022)
- [Machine Learning for Stuttering Identification: Review, Challenges and Future Directions](https://mlsanity.com/paper/2107.04057/) (2021)
- [From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition](https://mlsanity.com/paper/2302.09044/) (2023)

## Built on by (most cited first, 8 of 8)

- [Analysis and Evaluation of Synthetic Data Generation in Speech Dysfluency Detection](https://mlsanity.com/paper/2505.22029/) (2025)
- [Dysfluent WFST: A Framework for Zero-Shot Speech Dysfluency Transcription and Detection](https://mlsanity.com/paper/2505.16351/) (2025)
- [The Sound of Syntax: Finetuning and Comprehensive Evaluation of Language Models for Speech Pathology](https://mlsanity.com/paper/2509.16765/) (2025)
- [Smooth Operators: LLMs Translating Imperfect Hints into Disfluency-Rich Transcripts](https://mlsanity.com/paper/2506.18510/) (2025)
- [StutterCut: Uncertainty-Guided Normalised Cut for Dysfluency Segmentation](https://mlsanity.com/paper/2508.02255/) (2025)
- [Z-Scores: A Metric for Linguistically Assessing Disfluency Removal](https://mlsanity.com/paper/2509.20319/) (2025)
- [On the Difficulty of Token-Level Modeling of Dysfluency and Fluency Shaping Artifacts](https://mlsanity.com/paper/2512.02027/) (2025)
- [Mitigating Scoring Errors and Compensating for Nonverbal Subtests in Speech-Based Dementia Assessment](https://mlsanity.com/paper/2606.18979/) (2026)

## Abstract (arXiv)

Accurately detecting dysfluencies in spoken language can help to improve the performance of automatic speech and language processing components and support the development of more inclusive speech and language technologies. Inspired by the recent trend towards the deployment of large language models (LLMs) as universal learners and processors of non-lexical inputs, such as audio and video, we approach the task of multi-label dysfluency detection as a language modeling problem. We present hypotheses candidates generated with an automatic speech recognition system and acoustic representations extracted from an audio encoder model to an LLM, and finetune the system to predict dysfluency labels on three datasets containing English and German stuttered speech. The experimental results show that our system effectively combines acoustic and lexical information and achieves competitive results on the multi-label stuttering detection task.