# End-to-End Speech Recognition and Disfluency Removal

arXiv: https://arxiv.org/abs/2009.10298 · HTML: https://mlsanity.com/paper/2009.10298/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2020.

## Built on by (most cited first, 16 of 16)

- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [Disfluency Detection with Unlabeled Data and Small BERT Models](https://mlsanity.com/paper/2104.10769/) (2021)
- [Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection](https://mlsanity.com/paper/2205.00620/) (2022)
- [Streaming Joint Speech Recognition and Disfluency Detection](https://mlsanity.com/paper/2211.08726/) (2022)
- [Speculative End-Turn Detector for Efficient Speech Chatbot Assistant](https://mlsanity.com/paper/2503.23439/) (2025)
- [Fillers in Spoken Language Understanding: Computational and Psycholinguistic Perspectives](https://mlsanity.com/paper/2301.10761/) (2023)
- [Summarizing Speech: A Comprehensive Survey](https://mlsanity.com/paper/2504.08024/) (2025)
- [Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing](https://mlsanity.com/paper/2607.18934/) (2026)
- [Speech Technology Services for Oral History Research](https://mlsanity.com/paper/2405.02333/) (2024)
- [He Said, She Said: Style Transfer for Shifting the Perspective of Dialogues](https://mlsanity.com/paper/2210.15462/) (2022)
- [Artificial Disfluency Detection, Uh No, Disfluency Generation for the Masses](https://mlsanity.com/paper/2211.09235/) (2022)
- [Z-Scores: A Metric for Linguistically Assessing Disfluency Removal](https://mlsanity.com/paper/2509.20319/) (2025)
- [Mind the Pause: Disfluency-Aware Objective Tuning for Multilingual Speech Correction with LLMs](https://mlsanity.com/paper/2605.12242/) (2026)
- [End-to-End Speech Recognition and Disfluency Removal with Acoustic Language Model Pretraining](https://mlsanity.com/paper/2309.04516/) (2023)
- [Investigating large language models for their competence in extracting grammatically sound sentences from transcribed noisy utterances](https://mlsanity.com/paper/2410.05099/) (2024)
- [The Role of Disfluencies in Speech Translation](https://mlsanity.com/paper/2608.02138/) (2026)

## Abstract (arXiv)

Disfluency detection is usually an intermediate step between an automatic speech recognition (ASR) system and a downstream task. By contrast, this paper aims to investigate the task of end-to-end speech recognition and disfluency removal. We specifically explore whether it is possible to train an ASR model to directly map disfluent speech into fluent transcripts, without relying on a separate disfluency detection model. We show that end-to-end models do learn to directly generate fluent transcripts; however, their performance is slightly worse than a baseline pipeline approach consisting of an ASR system and a disfluency detection model. We also propose two new metrics that can be used for evaluating integrated ASR and disfluency models. The findings of this paper can serve as a benchmark for further research on the task of end-to-end speech recognition and disfluency removal in the future.