# Stutter-TTS: Controlled Synthesis and Improved Recognition of Stuttered Speech

arXiv: https://arxiv.org/abs/2211.09731 · HTML: https://mlsanity.com/paper/2211.09731/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 4 Nov 2022.

## Builds on (most cited first, 12 of 12)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation](https://mlsanity.com/paper/2204.01171/) (2022)
- [SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stutter](https://mlsanity.com/paper/2102.12394/) (2021)
- [MultiSpeech: Multi-Speaker Text to Speech with Transformer](https://mlsanity.com/paper/2006.04664/) (2020)
- [Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systems](https://mlsanity.com/paper/2011.11564/) (2020)
- [Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0](https://mlsanity.com/paper/2204.03417/) (2022)
- [Controllable Context-aware Conversational Speech Synthesis](https://mlsanity.com/paper/2106.10828/) (2021)
- [Speaker Generation](https://mlsanity.com/paper/2111.05095/) (2021)
- [Synthesizing Dysarthric Speech Using Multi-talker TTS for Dysarthric Speech Recognition](https://mlsanity.com/paper/2201.11571/) (2022)

## Built on by (most cited first, 3 of 3)

- [Machine Learning for Synthetic Data Generation: A Review](https://mlsanity.com/paper/2302.04062/) (2023)
- [Synthetic Data in AI: Challenges, Applications, and Ethical Implications](https://mlsanity.com/paper/2401.01629/) (2024)
- [AS-70: A Mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection](https://mlsanity.com/paper/2406.07256/) (2024)

## Abstract (arXiv)

Stuttering is a speech disorder where the natural flow of speech is interrupted by blocks, repetitions or prolongations of syllables, words and phrases. The majority of existing automatic speech recognition (ASR) interfaces perform poorly on utterances with stutter, mainly due to lack of matched training data. Synthesis of speech with stutter thus presents an opportunity to improve ASR for this type of speech. We describe Stutter-TTS, an end-to-end neural text-to-speech model capable of synthesizing diverse types of stuttering utterances. We develop a simple, yet effective prosody-control strategy whereby additional tokens are introduced into source text during training to represent specific stuttering characteristics. By choosing the position of the stutter tokens, Stutter-TTS allows word-level control of where stuttering occurs in the synthesized utterance. We are able to synthesize stutter events with high accuracy (F1-scores between 0.63 and 0.84, depending on stutter type). By fine-tuning an ASR model on synthetic stuttered speech we are able to reduce word error by 5.7% relative on stuttered utterances, with only minor (<0.2% relative) degradation for fluent utterances.