# SSDM: Scalable Speech Dysfluency Modeling

arXiv: https://arxiv.org/abs/2408.16221 · HTML: https://mlsanity.com/paper/2408.16221/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 85 of its 139 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 30 of 85)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)

## Built on by (most cited first, 13 of 13)

- [Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models](https://mlsanity.com/paper/2505.03075/) (2025)
- [Analysis and Evaluation of Synthetic Data Generation in Speech Dysfluency Detection](https://mlsanity.com/paper/2505.22029/) (2025)
- [Whisper Has an Internal Word Aligner](https://mlsanity.com/paper/2509.09987/) (2025)
- [Dysfluent WFST: A Framework for Zero-Shot Speech Dysfluency Transcription and Detection](https://mlsanity.com/paper/2505.16351/) (2025)
- [Seamless Dysfluent Speech Text Alignment for Disordered Speech Analysis](https://mlsanity.com/paper/2506.12073/) (2025)
- [HuPER: A Human-Inspired Framework for Phonetic Perception](https://mlsanity.com/paper/2602.01634/) (2026)
- [K-Function: Joint Pronunciation Transcription and Feedback for Evaluating Kids Language Function](https://mlsanity.com/paper/2507.03043/) (2025)
- [Enabling Conversational Behavior Reasoning Capabilities in Full-Duplex Speech](https://mlsanity.com/paper/2512.21706/) (2025)
- [Revisiting Rule-Based Stuttering Detection: A Comprehensive Analysis of Interpretable Models for Clinical Applications](https://mlsanity.com/paper/2508.16681/) (2025)
- [A Comparative Study of Controllability, Explainability, and Performance in Dysfluency Detection Models](https://mlsanity.com/paper/2509.00058/) (2025)
- [Deploying UDM Series in Real-Life Stuttered Speech Applications: A Clinical Evaluation Framework](https://mlsanity.com/paper/2509.14304/) (2025)
- [SpikeVox: Towards Energy-Efficient Speech Therapy Framework with Spike-driven Generative Language Models](https://mlsanity.com/paper/2510.15566/) (2025)
- [HASS: Hierarchical Simulation of Logopenic Aphasic Speech for Scalable PPA Detection](https://mlsanity.com/paper/2603.26795/) (2026)

## Abstract (arXiv)

Speech dysfluency modeling is the core module for spoken language learning, and speech therapy. However, there are three challenges. First, current state-of-the-art solutionscitelian2023unconstrained-udm, lian-anumanchipalli-2024-towards-hudm suffer from poor scalability. Second, there is a lack of a large-scale dysfluency corpus. Third, there is not an effective learning framework. In this paper, we propose textitSSDM: Scalable Speech Dysfluency Modeling, which (1) adopts articulatory gestures as scalable forced alignment; (2) introduces connectionist subsequence aligner (CSA) to achieve dysfluency alignment; (3) introduces a large-scale simulated dysfluency corpus called Libri-Dys; and (4) develops an end-to-end system by leveraging the power of large language models (LLMs). We expect SSDM to serve as a standard in the area of dysfluency modeling. Demo is available at urlhttps://berkeley-speech-group.github.io/SSDM/.