# Disfluency Detection with Unlabeled Data and Small BERT Models

arXiv: https://arxiv.org/abs/2104.10769 · HTML: https://mlsanity.com/paper/2104.10769/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 21 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [Disfluency Detection using a Bidirectional LSTM](https://mlsanity.com/paper/1604.03209/) (2016)
- [Improving Disfluency Detection by Self-Training a Self-Attentive Model](https://mlsanity.com/paper/2004.05323/) (2020)
- [Disfluencies and Human Speech Transcription Errors](https://mlsanity.com/paper/1904.04398/) (2019)
- [End-to-End Speech Recognition and Disfluency Removal](https://mlsanity.com/paper/2009.10298/) (2020)
- [Disfluency Detection using Auto-Correlational Neural Networks](https://mlsanity.com/paper/1808.09092/) (2018)
- [Disfluency Detection using a Noisy Channel Model and a Deep Neural Language Model](https://mlsanity.com/paper/1808.09091/) (2018)
- [FluentNet: End-to-End Detection of Speech Disfluency with Deep Learning](https://mlsanity.com/paper/2009.11394/) (2020)
- [Robust cross-domain disfluency detection with pattern match networks](https://mlsanity.com/paper/1811.07236/) (2018)

## Built on by (most cited first, 17 of 17)

- [Turn-Taking Prediction for Natural Conversational Speech](https://mlsanity.com/paper/2208.13321/) (2022)
- [Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection](https://mlsanity.com/paper/2205.00620/) (2022)
- [LARD: Large-scale Artificial Disfluency Generation](https://mlsanity.com/paper/2201.05041/) (2022)
- [Streaming Joint Speech Recognition and Disfluency Detection](https://mlsanity.com/paper/2211.08726/) (2022)
- [Automatic Disfluency Detection from Untranscribed Speech](https://mlsanity.com/paper/2311.00867/) (2023)
- [Augmenting Automatic Speech Recognition Models with Disfluency Detection](https://mlsanity.com/paper/2409.10177/) (2024)
- [Missingness-resilient Video-enhanced Multimodal Disfluency Detection](https://mlsanity.com/paper/2406.06964/) (2024)
- [Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion](https://mlsanity.com/paper/2412.12710/) (2024)
- [From Disfluency Detection to Intent Detection and Slot Filling](https://mlsanity.com/paper/2209.08359/) (2022)
- [Transcription free filler word detection with Neural semi-CRFs](https://mlsanity.com/paper/2303.06475/) (2023)
- [MultiTurnCleanup: A Benchmark for Multi-Turn Spoken Conversational Transcript Cleanup](https://mlsanity.com/paper/2305.12029/) (2023)
- [Boosting Disfluency Detection with Large Language Model as Disfluency Generator](https://mlsanity.com/paper/2403.08229/) (2024)
- [Artificial Disfluency Detection, Uh No, Disfluency Generation for the Masses](https://mlsanity.com/paper/2211.09235/) (2022)
- [Humane Speech Synthesis through Zero-Shot Emotion and Disfluency Generation](https://mlsanity.com/paper/2404.01339/) (2024)
- [Adapting the NICT-JLE Corpus for Disfluency Detection Models](https://mlsanity.com/paper/2308.02482/) (2023)
- [End-to-End Speech Recognition and Disfluency Removal with Acoustic Language Model Pretraining](https://mlsanity.com/paper/2309.04516/) (2023)
- [On the Difficulty of Token-Level Modeling of Dysfluency and Fluency Shaping Artifacts](https://mlsanity.com/paper/2512.02027/) (2025)

## Abstract (arXiv)

Disfluency detection models now approach high accuracy on English text. However, little exploration has been done in improving the size and inference time of the model. At the same time, automatic speech recognition (ASR) models are moving from server-side inference to local, on-device inference. Supporting models in the transcription pipeline (like disfluency detection) must follow suit. In this work we concentrate on the disfluency detection task, focusing on small, fast, on-device models based on the BERT architecture. We demonstrate it is possible to train disfluency detection models as small as 1.3 MiB, while retaining high performance. We build on previous work that showed the benefit of data augmentation approaches such as self-training. Then, we evaluate the effect of domain mismatch between conversational and written text on model performance. We find that domain adaptation and data augmentation strategies have a more pronounced effect on these smaller models, as compared to conventional BERT models.