# Whisper in Focus: Enhancing Stuttered Speech Classification with Encoder Layer Optimization

arXiv: https://arxiv.org/abs/2311.05203 · HTML: https://mlsanity.com/paper/2311.05203/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 10 of 10)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stutter](https://mlsanity.com/paper/2102.12394/) (2021)
- [Improved DeepFake Detection Using Whisper Features](https://mlsanity.com/paper/2306.01428/) (2023)
- [Detecting Multiple Speech Disfluencies using a Deep Residual Network with Bidirectional Long Short-Term Memory](https://mlsanity.com/paper/1910.12590/) (2019)
- [Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0](https://mlsanity.com/paper/2204.03417/) (2022)
- [KSoF: The Kassel State of Fluency Dataset -- A Therapy Centered Dataset of Stuttering](https://mlsanity.com/paper/2203.05383/) (2022)
- [StutterNet: Stuttering Detection Using Time Delay Neural Network](https://mlsanity.com/paper/2105.05599/) (2021)
- [Machine Learning for Stuttering Identification: Review, Challenges and Future Directions](https://mlsanity.com/paper/2107.04057/) (2021)
- [The Influence of Dataset Partitioning on Dysfluency Detection Systems](https://mlsanity.com/paper/2206.03400/) (2022)

## Built on by (most cited first, 5 of 5)

- [Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whisper](https://mlsanity.com/paper/2406.05806/) (2024)
- [Classification of Spontaneous and Scripted Speech for Multilingual Audio](https://mlsanity.com/paper/2412.11896/) (2024)
- [Optimizing Multi-Stuttered Speech Classification: Leveraging Whisper's Encoder for Efficient Parameter Reduction in Automated Assessment](https://mlsanity.com/paper/2406.05784/) (2024)
- [A Semi-Supervised Framework for Speech Confidence Detection using Whisper](https://mlsanity.com/paper/2605.12387/) (2026)
- [Semi-Supervised Speech Confidence Detection using Pseudo-Labelling and Whisper Embeddings](https://mlsanity.com/paper/2606.16505/) (2026)

## Abstract (arXiv)

In recent years, advancements in the field of speech processing have led to cutting-edge deep learning algorithms with immense potential for real-world applications. The automated identification of stuttered speech is one of such applications that the researchers are addressing by employing deep learning techniques. Recently, researchers have utilized Wav2vec2.0, a speech recognition model to classify disfluency types in stuttered speech. Although Wav2vec2.0 has shown commendable results, its ability to generalize across all disfluency types is limited. In addition, since its base model uses 12 encoder layers, it is considered a resource-intensive model. Our study unravels the capabilities of Whisper for the classification of disfluency types in stuttered speech. We have made notable contributions in three pivotal areas: enhancing the quality of SEP28-k benchmark dataset, exploration of Whisper for classification, and introducing an efficient encoder layer freezing strategy. The optimized Whisper model has achieved the average F1-score of 0.81, which proffers its abilities. This study also unwinds the significance of deeper encoder layers in the identification of disfluency types, as the results demonstrate their greater contribution compared to initial layers. This research represents substantial contributions, shifting the emphasis towards an efficient solution, thereby thriving towards prospective innovation.