# Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR

arXiv: https://arxiv.org/abs/2606.14391 · HTML: https://mlsanity.com/paper/2606.14391/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 12 Jun 2026.

## Builds on (most cited first, 17 of 17)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Continual Learning and Catastrophic Forgetting](https://mlsanity.com/paper/2403.05175/) (2024)
- [LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR](https://mlsanity.com/paper/2406.06619/) (2024)
- [CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions](https://mlsanity.com/paper/2408.16589/) (2024)
- [CL-MASR: A Continual Learning Benchmark for Multilingual ASR](https://mlsanity.com/paper/2310.16931/) (2023)
- [Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech](https://mlsanity.com/paper/2405.06150/) (2024)
- [Rehearsal-Free Online Continual Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2306.10860/) (2023)
- [Replay to Remember: Continual Layer-Specific Fine-tuning for German Speech Recognition](https://mlsanity.com/paper/2307.07280/) (2023)
- [Visually Grounded Continual Language Learning with Selective Specialization](https://mlsanity.com/paper/2310.15571/) (2023)
- [WhisperD: Dementia Speech Recognition and Filler Word Detection with Whisper](https://mlsanity.com/paper/2505.21551/) (2025)
- [Acoustically Precise Hesitation Tagging Is Essential for End-to-End Verbatim Transcription Systems](https://mlsanity.com/paper/2506.04076/) (2025)
- [Talking to Robots: A Practical Examination of Speech Foundation Models for HRI Applications](https://mlsanity.com/paper/2508.17753/) (2025)

## Abstract (arXiv)

Despite advances in large-scale Automatic Speech Recognition (ASR), disfluent speech remains challenging, as state-of-the-art systems are often optimized to omit disfluencies, leading to information loss and hallucinations. Prior work has focused on verbatim transcription and the integration of disfluency markers, but adapting models on limited datasets can lead to catastrophic forgetting of general-domain knowledge. We address this gap by leveraging continual learning (CL) with explicit disfluency tokens. We first introduce these tokens into a pretrained ASR model to establish stable token mechanisms, and then continue training on additional datasets with varying disfluency distributions. Through a detailed analysis of model dynamics during training, we identify a trade-off between marker learning and ASR performance, and a consistent cross-attention head mechanism shared across CL methods.