# Inclusive ASR for Disfluent Speech: Cascaded Large-Scale Self-Supervised Learning with Targeted Fine-Tuning and Data Augmentation

arXiv: https://arxiv.org/abs/2406.10177 · HTML: https://mlsanity.com/paper/2406.10177/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 13 of 13)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stutter](https://mlsanity.com/paper/2102.12394/) (2021)
- [Speaker adaptation for Wav2vec2 based dysarthric ASR](https://mlsanity.com/paper/2204.00770/) (2022)
- [Machine Learning for Stuttering Identification: Review, Challenges and Future Directions](https://mlsanity.com/paper/2107.04057/) (2021)
- [From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition](https://mlsanity.com/paper/2302.09044/) (2023)
- [Personalized Automatic Speech Recognition Trained on Small Disordered Speech Datasets](https://mlsanity.com/paper/2110.04612/) (2021)
- [Analysis and Tuning of a Voice Assistant System for Dysfluent Speech](https://mlsanity.com/paper/2106.11759/) (2021)
- [FluentNet: End-to-End Detection of Speech Disfluency with Deep Learning](https://mlsanity.com/paper/2009.11394/) (2020)
- [Assessing ASR Model Quality on Disordered Speech using BERTScore](https://mlsanity.com/paper/2209.10591/) (2022)

## Built on by (most cited first, 5 of 5)

- [Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing](https://mlsanity.com/paper/2607.18934/) (2026)
- [On the Difficulty of Token-Level Modeling of Dysfluency and Fluency Shaping Artifacts](https://mlsanity.com/paper/2512.02027/) (2025)
- [Stuttering-Aware Automatic Speech Recognition for Indonesian Language](https://mlsanity.com/paper/2601.03727/) (2026)
- [Unheard in the Digital Age: Rethinking AI Bias and Speech Diversity](https://mlsanity.com/paper/2601.18641/) (2026)
- [Aligning Stuttered-Speech Research with End-User Needs: Scoping Review, Survey, and Guidelines](https://mlsanity.com/paper/2604.20535/) (2026)

## Abstract (arXiv)

Automatic speech recognition (ASR) systems often falter while processing stuttering-related disfluencies -- such as involuntary blocks and word repetitions -- yielding inaccurate transcripts. A critical barrier to progress is the scarcity of large, annotated disfluent speech datasets. Therefore, we present an inclusive ASR design approach, leveraging large-scale self-supervised learning on standard speech followed by targeted fine-tuning and data augmentation on a smaller, curated dataset of disfluent speech. Our data augmentation technique enriches training datasets with various disfluencies, enhancing ASR processing of these speech patterns. Results show that fine-tuning wav2vec 2.0 with even a relatively small, labeled dataset, alongside data augmentation, can significantly reduce word error rates for disfluent speech. Our approach not only advances ASR inclusivity for people who stutter, but also paves the way for ASRs that can accommodate wider speech variations.