# On the Relevance of Phoneme Duration Variability of Synthesized Training Data for Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2310.08132 · HTML: https://mlsanity.com/paper/2310.08132/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [FastPitch: Parallel Text-to-speech with Pitch Prediction](https://mlsanity.com/paper/2006.06873/) (2020)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [Speech Recognition with Augmented Synthesized Speech](https://mlsanity.com/paper/1909.11699/) (2019)
- [The VoiceMOS Challenge 2022](https://mlsanity.com/paper/2203.11389/) (2022)
- [Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems](https://mlsanity.com/paper/1912.09257/) (2019)
- [Exploring Neural Transducers for End-to-End Speech Recognition](https://mlsanity.com/paper/1707.07413/) (2017)
- [Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systems](https://mlsanity.com/paper/2011.11564/) (2020)
- [Improved training of end-to-end attention models for speech recognition](https://mlsanity.com/paper/1805.03294/) (2018)
- [You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation](https://mlsanity.com/paper/2005.07157/) (2020)
- [Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition](https://mlsanity.com/paper/2110.11479/) (2021)
- [AutoMOS: Learning a non-intrusive assessor of naturalness-of-speech](https://mlsanity.com/paper/1611.09207/) (2016)
- [RETURNN: The RWTH Extensible Training framework for Universal Recurrent Neural Networks](https://mlsanity.com/paper/1608.00895/) (2016)
- [EAT: Enhanced ASR-TTS for Self-supervised Speech Recognition](https://mlsanity.com/paper/2104.07474/) (2021)
- [Evaluating and reducing the distance between synthetic and real speech distributions](https://mlsanity.com/paper/2211.16049/) (2022)
- [Comparing the Benefit of Synthetic Training Data for Various Automatic Speech Recognition Architectures](https://mlsanity.com/paper/2104.05379/) (2021)
- [VRAIN-UPV MLLP's system for the Blizzard Challenge 2021](https://mlsanity.com/paper/2110.15792/) (2021)

## Built on by (most cited first, 4 of 4)

- [An Exhaustive Evaluation of TTS- and VC-based Data Augmentation for ASR](https://mlsanity.com/paper/2503.08954/) (2025)
- [On the Effect of Purely Synthetic Training Data for Different Automatic Speech Recognition Architectures](https://mlsanity.com/paper/2407.17997/) (2024)
- [On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition](https://mlsanity.com/paper/2407.21476/) (2024)
- [Continuous-Time Acoustic Modelling with Neural Controlled Differential Equations](https://mlsanity.com/paper/2609.11725/) (2026)

## Abstract (arXiv)

Synthetic data generated by text-to-speech (TTS) systems can be used to improve automatic speech recognition (ASR) systems in low-resource or domain mismatch tasks. It has been shown that TTS-generated outputs still do not have the same qualities as real data. In this work we focus on the temporal structure of synthetic data and its relation to ASR training. By using a novel oracle setup we show how much the degradation of synthetic data quality is influenced by duration modeling in non-autoregressive (NAR) TTS. To get reference phoneme durations we use two common alignment methods, a hidden Markov Gaussian-mixture model (HMM-GMM) aligner and a neural connectionist temporal classification (CTC) aligner. Using a simple algorithm based on random walks we shift phoneme duration distributions of the TTS system closer to real durations, resulting in an improvement of an ASR system using synthetic data in a semi-supervised setting.