# Optimizing Multi-Stuttered Speech Classification: Leveraging Whisper's Encoder for Efficient Parameter Reduction in Automated Assessment

arXiv: https://arxiv.org/abs/2406.05784 · HTML: https://mlsanity.com/paper/2406.05784/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 9 Jun 2024.

## Builds on (most cited first, 13 of 13)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stutter](https://mlsanity.com/paper/2102.12394/) (2021)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0](https://mlsanity.com/paper/2204.03417/) (2022)
- [Vision Transformers in Medical Computer Vision -- A Contemplative Retrospection](https://mlsanity.com/paper/2203.15269/) (2022)
- [KSoF: The Kassel State of Fluency Dataset -- A Therapy Centered Dataset of Stuttering](https://mlsanity.com/paper/2203.05383/) (2022)
- [StutterNet: Stuttering Detection Using Time Delay Neural Network](https://mlsanity.com/paper/2105.05599/) (2021)
- [Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretraining](https://mlsanity.com/paper/2301.07295/) (2023)
- [The Influence of Dataset Partitioning on Dysfluency Detection Systems](https://mlsanity.com/paper/2206.03400/) (2022)
- [A Stutter Seldom Comes Alone -- Cross-Corpus Stuttering Detection as a Multi-label Problem](https://mlsanity.com/paper/2305.19255/) (2023)
- [Whisper in Focus: Enhancing Stuttered Speech Classification with Encoder Layer Optimization](https://mlsanity.com/paper/2311.05203/) (2023)
- [Dysfluencies Seldom Come Alone -- Detection as a Multi-Label Problem](https://mlsanity.com/paper/2210.15982/) (2022)

## Abstract (arXiv)

The automated classification of stuttered speech has significant implications for timely assessments providing assistance to speech language pathologists. Despite notable advancements in the field, the cases in which multiple disfluencies occur in speech require attention. We have taken a progressive approach to fill this gap by classifying multi-stuttered speech more efficiently. The problem has been addressed by firstly curating a dataset of multi-stuttered disfluencies from open source dataset SEP-28k audio clips. Secondly, employing Whisper, a state-of-the-art speech recognition model has been leveraged by using its encoder and taking the problem as multi label classification. Thirdly, using a 6 encoder layer Whisper and experimenting with various layer freezing strategies, a computationally efficient configuration of the model was identified. The proposed configuration achieved micro, macro, and weighted F1-scores of 0.88, 0.85, and 0.87, correspondingly on an external test dataset i.e. Fluency-Bank. In addition, through layer freezing strategies, we were able to achieve the aforementioned results by fine-tuning a single encoder layer, consequently, reducing the model's trainable parameters from 20.27 million to 3.29 million. This research study unveils the contribution of the last encoder layer in the identification of disfluencies in stuttered speech. Consequently, it has led to a computationally efficient approach, 83.7% less parameters to train, making the proposed approach more adaptable for various dialects and languages.