# StutterZero and StutterFormer: End-to-End Speech Conversion for Stuttering Transcription and Correction

arXiv: https://arxiv.org/abs/2510.18938 · HTML: https://mlsanity.com/paper/2510.18938/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [An Overview of Affective Speech Synthesis and Conversion in the Deep Learning Era](https://mlsanity.com/paper/2210.03538/) (2022)
- [Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition](https://mlsanity.com/paper/1704.01631/) (2017)
- [StutterNet: Stuttering Detection Using Time Delay Neural Network](https://mlsanity.com/paper/2105.05599/) (2021)
- [Machine Learning for Stuttering Identification: Review, Challenges and Future Directions](https://mlsanity.com/paper/2107.04057/) (2021)
- [From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition](https://mlsanity.com/paper/2302.09044/) (2023)
- [On loss functions and evaluation metrics for music source separation](https://mlsanity.com/paper/2202.07968/) (2022)
- [Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentation](https://mlsanity.com/paper/2401.00662/) (2024)
- [FluentNet: End-to-End Detection of Speech Disfluency with Deep Learning](https://mlsanity.com/paper/2009.11394/) (2020)
- [AS-70: A Mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection](https://mlsanity.com/paper/2406.07256/) (2024)
- [From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling](https://mlsanity.com/paper/2510.00743/) (2025)
- [CADE: Cosine Annealing Differential Evolution for Spiking Neural Network](https://mlsanity.com/paper/2406.02349/) (2024)
- [Fine-Tuning ASR for Stuttered Speech: Personalized vs. Generalized Approaches](https://mlsanity.com/paper/2506.00853/) (2025)
- [Fluent: An AI Augmented Writing Tool for People who Stutter](https://mlsanity.com/paper/2108.09918/) (2021)
- [Loss functions incorporating auditory spatial perception in deep learning -- a review](https://mlsanity.com/paper/2506.19404/) (2025)

## Built on by (most cited first, 1 of 1)

- [Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing](https://mlsanity.com/paper/2607.18934/) (2026)

## Abstract (arXiv)

Over 70 million people worldwide experience stuttering, yet most automatic speech systems misinterpret disfluent utterances or fail to transcribe them accurately. Existing methods for stutter correction rely on handcrafted feature extraction or multi-stage automatic speech recognition (ASR) and text-to-speech (TTS) pipelines, which separate transcription from audio reconstruction and often amplify distortions. This work introduces StutterZero and StutterFormer, the first end-to-end waveform-to-waveform models that directly convert stuttered speech into fluent speech while jointly predicting its transcription. StutterZero employs a convolutional-bidirectional LSTM encoder-decoder with attention, whereas StutterFormer integrates a dual-stream Transformer with shared acoustic-linguistic representations. Both architectures are trained on paired stuttered-fluent data synthesized from the SEP-28K and LibriStutter corpora and evaluated on unseen speakers from the FluencyBank dataset. Across all benchmarks, StutterZero had a 24% decrease in Word Error Rate (WER) and a 31% improvement in semantic similarity (BERTScore) compared to the leading Whisper-Medium model. StutterFormer achieved better results, with a 28% decrease in WER and a 34% improvement in BERTScore. The results validate the feasibility of direct end-to-end stutter-to-fluent speech conversion, offering new opportunities for inclusive human-computer interaction, speech therapy, and accessibility-oriented AI systems.