# On the Difficulty of Token-Level Modeling of Dysfluency and Fluency Shaping Artifacts

arXiv: https://arxiv.org/abs/2512.02027 · HTML: https://mlsanity.com/paper/2512.02027/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 18 Nov 2025.

## Builds on (most cited first, 19 of 19)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [ASR Error Correction using Large Language Models](https://mlsanity.com/paper/2409.09554/) (2024)
- [KSoF: The Kassel State of Fluency Dataset -- A Therapy Centered Dataset of Stuttering](https://mlsanity.com/paper/2203.05383/) (2022)
- [Unconstrained Dysfluency Modeling for Dysfluent Speech Transcription and Detection](https://mlsanity.com/paper/2312.12810/) (2023)
- [Disfluency Detection with Unlabeled Data and Small BERT Models](https://mlsanity.com/paper/2104.10769/) (2021)
- [Personalized Automatic Speech Recognition Trained on Small Disordered Speech Datasets](https://mlsanity.com/paper/2110.04612/) (2021)
- [Adapting an ASR Foundation Model for Spoken Language Assessment](https://mlsanity.com/paper/2307.09378/) (2023)
- [Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection](https://mlsanity.com/paper/2205.00620/) (2022)
- [Large Language Models for Dysfluency Detection in Stuttered Speech](https://mlsanity.com/paper/2406.11025/) (2024)
- [Studying Alignment in a Collaborative Learning Activity via Automatic Methods: The Link Between What We Say and Do](https://mlsanity.com/paper/2104.04429/) (2021)
- [Streaming Joint Speech Recognition and Disfluency Detection](https://mlsanity.com/paper/2211.08726/) (2022)
- [Augmenting Automatic Speech Recognition Models with Disfluency Detection](https://mlsanity.com/paper/2409.10177/) (2024)
- [Inclusive ASR for Disfluent Speech: Cascaded Large-Scale Self-Supervised Learning with Targeted Fine-Tuning and Data Augmentation](https://mlsanity.com/paper/2406.10177/) (2024)
- [A Parameter-efficient Language Extension Framework for Multilingual ASR](https://mlsanity.com/paper/2406.06329/) (2024)

## Abstract (arXiv)

Automatic transcription of stuttered speech remains a challenge, even for modern end-to-end (E2E) automatic speech recognition (ASR) frameworks. Dysfluencies and fluency-shaping artifacts are often overlooked, resulting in non-verbatim transcriptions with limited clinical and research value. We propose a parameter-efficient adaptation method to decode dysfluencies and fluency modifications as special tokens within transcriptions, evaluated on simulated (LibriStutter, English) and natural (KSoF, German) stuttered speech datasets. To mitigate ASR performance disparities and bias towards English, we introduce a multi-step fine-tuning strategy with language-adaptive pretraining. Tokenization analysis further highlights the tokenizer's English-centric bias, which poses challenges for improving performance on German data. Our findings demonstrate the effectiveness of lightweight adaptation techniques for dysfluency-aware ASR while exposing key limitations in multilingual E2E systems.