# Filler Word Detection and Classification: A Dataset and Benchmark

arXiv: https://arxiv.org/abs/2203.15135 · HTML: https://mlsanity.com/paper/2203.15135/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 10 of 10)

- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [The Benefit Of Temporally-Strong Labels In Audio Event Classification](https://mlsanity.com/paper/2105.07031/) (2021)
- [SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stutter](https://mlsanity.com/paper/2102.12394/) (2021)
- [Temporal Convolution for Real-time Keyword Spotting on Mobile Devices](https://mlsanity.com/paper/1904.03814/) (2019)
- [Multi-Task Self-Supervised Learning for Disfluency Detection](https://mlsanity.com/paper/1908.05378/) (2019)
- [StutterNet: Stuttering Detection Using Time Delay Neural Network](https://mlsanity.com/paper/2105.05599/) (2021)
- [Disfluency Detection using Auto-Correlational Neural Networks](https://mlsanity.com/paper/1808.09092/) (2018)
- [Giving Attention to the Unexpected: Using Prosody Innovations in Disfluency Detection](https://mlsanity.com/paper/1904.04388/) (2019)
- [Increase Apparent Public Speaking Fluency By Speech Augmentation](https://mlsanity.com/paper/1812.03415/) (2018)

## Built on by (most cited first, 7 of 7)

- [CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions](https://mlsanity.com/paper/2408.16589/) (2024)
- [Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation](https://mlsanity.com/paper/2408.11849/) (2024)
- [Fillers in Spoken Language Understanding: Computational and Psycholinguistic Perspectives](https://mlsanity.com/paper/2301.10761/) (2023)
- [Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing](https://mlsanity.com/paper/2607.18934/) (2026)
- [Transcription free filler word detection with Neural semi-CRFs](https://mlsanity.com/paper/2303.06475/) (2023)
- [WhisperD: Dementia Speech Recognition and Filler Word Detection with Whisper](https://mlsanity.com/paper/2505.21551/) (2025)
- [SEAM: Shortcut-Aware Real-Time Detection of Scripted vs. Spontaneous Speech for Interview Guardrails](https://mlsanity.com/paper/2606.06837/) (2026)

## Abstract (arXiv)

Filler words such as `uh' or `um' are sounds or words people use to signal they are pausing to think. Finding and removing filler words from recordings is a common and tedious task in media editing. Automatically detecting and classifying filler words could greatly aid in this task, but few studies have been published on this problem to date. A key reason is the absence of a dataset with annotated filler words for model training and evaluation. In this work, we present a novel speech dataset, PodcastFillers, with 35K annotated filler words and 50K annotations of other sounds that commonly occur in podcasts such as breaths, laughter, and word repetitions. We propose a pipeline that leverages VAD and ASR to detect filler candidates and a classifier to distinguish between filler word types. We evaluate our proposed pipeline on PodcastFillers, compare to several baselines, and present a detailed ablation study. In particular, we evaluate the importance of using ASR and how it compares to a transcription-free approach resembling keyword spotting. We show that our pipeline obtains state-of-the-art results, and that leveraging ASR strongly outperforms a keyword spotting approach. We make PodcastFillers publicly available, in the hope that our work serves as a benchmark for future research.