# Advancing Stuttering Detection via Data Augmentation, Class-Balanced Loss and Multi-Contextual Deep Learning

arXiv: https://arxiv.org/abs/2302.11343 · HTML: https://mlsanity.com/paper/2302.11343/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 70 references that are in the map.
- First posted on arXiv 21 Feb 2023.

## Builds on (most cited first, 21 of 21)

- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [An Overview of Voice Conversion and its Challenges: From Statistical Modeling to Deep Learning](https://mlsanity.com/paper/2008.03648/) (2020)
- [SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stutter](https://mlsanity.com/paper/2102.12394/) (2021)
- [MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognition](https://mlsanity.com/paper/2102.12664/) (2021)
- [Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual Speech Recognition](https://mlsanity.com/paper/2012.01687/) (2020)
- [Disfluency Detection using a Bidirectional LSTM](https://mlsanity.com/paper/1604.03209/) (2016)
- [Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends](https://mlsanity.com/paper/2001.00378/) (2020)
- [Detecting Multiple Speech Disfluencies using a Deep Residual Network with Bidirectional Long Short-Term Memory](https://mlsanity.com/paper/1910.12590/) (2019)
- [Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detection](https://mlsanity.com/paper/2003.01309/) (2020)
- [StutterNet: Stuttering Detection Using Time Delay Neural Network](https://mlsanity.com/paper/2105.05599/) (2021)
- [Machine Learning for Stuttering Identification: Review, Challenges and Future Directions](https://mlsanity.com/paper/2107.04057/) (2021)
- [Simulating dysarthric speech for training data augmentation in clinical speech applications](https://mlsanity.com/paper/1804.10325/) (2018)
- [Multistream CNN for Robust Acoustic Modeling](https://mlsanity.com/paper/2005.10470/) (2020)
- [Analysis and Tuning of a Voice Assistant System for Dysfluent Speech](https://mlsanity.com/paper/2106.11759/) (2021)
- [Data Augmentation for Dementia Detection in Spoken Language](https://mlsanity.com/paper/2206.12879/) (2022)
- [End-to-End and Self-Supervised Learning for ComParE 2022 Stuttering Sub-Challenge](https://mlsanity.com/paper/2207.10817/) (2022)
- [The ACM Multimedia 2022 Computational Paralinguistics Challenge: Vocalisations, Stuttering, Activity, & Mosquitoes](https://mlsanity.com/paper/2205.06799/) (2022)
- [Temporal envelope and fine structure cues for dysarthric speech detection using CNNs](https://mlsanity.com/paper/2108.11153/) (2021)

## Built on by (most cited first, 5 of 5)

- [AS-70: A Mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection](https://mlsanity.com/paper/2406.07256/) (2024)
- [Stuttering Detection Using Speaker Representations and Self-supervised Contextual Embeddings](https://mlsanity.com/paper/2306.00689/) (2023)
- [Impact of Speech Mode in Automatic Pathological Speech Detection](https://mlsanity.com/paper/2406.09968/) (2024)
- [Selfsupervised learning for pathological speech detection](https://mlsanity.com/paper/2406.02572/) (2024)
- [Virtual Speech Therapist: A Clinician-in-the-Loop AI Speech Therapy Agent for Personalized and Supervised Therapy](https://mlsanity.com/paper/2605.01101/) (2026)

## Abstract (arXiv)

Stuttering is a neuro-developmental speech impairment characterized by uncontrolled utterances (interjections) and core behaviors (blocks, repetitions, and prolongations), and is caused by the failure of speech sensorimotors. Due to its complex nature, stuttering detection (SD) is a difficult task. If detected at an early stage, it could facilitate speech therapists to observe and rectify the speech patterns of persons who stutter (PWS). The stuttered speech of PWS is usually available in limited amounts and is highly imbalanced. To this end, we address the class imbalance problem in the SD domain via a multibranching (MB) scheme and by weighting the contribution of classes in the overall loss function, resulting in a huge improvement in stuttering classes on the SEP-28k dataset over the baseline (StutterNet). To tackle data scarcity, we investigate the effectiveness of data augmentation on top of a multi-branched training scheme. The augmented training outperforms the MB StutterNet (clean) by a relative margin of 4.18% in macro F1-score (F1). In addition, we propose a multi-contextual (MC) StutterNet, which exploits different contexts of the stuttered speech, resulting in an overall improvement of 4.48% in F 1 over the single context based MB StutterNet. Finally, we have shown that applying data augmentation in the cross-corpora scenario can improve the overall SD performance by a relative margin of 13.23% in F1 over the clean training.