# SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models

arXiv: https://arxiv.org/abs/2509.14270 · HTML: https://mlsanity.com/paper/2509.14270/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Builds on (most cited first, 19 of 19)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [Diversity in Machine Learning](https://mlsanity.com/paper/1807.01477/) (2018)
- [Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration](https://mlsanity.com/paper/2109.06304/) (2021)
- [NeMo Inverse Text Normalization: From Development To Production](https://mlsanity.com/paper/2104.05055/) (2021)
- [Diversity of Thought Improves Reasoning Abilities of LLMs](https://mlsanity.com/paper/2310.07088/) (2023)
- [Phoneme-Level BERT for Enhanced Prosody of Text-to-Speech with Grapheme Predictions](https://mlsanity.com/paper/2301.08810/) (2023)
- [SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use](https://mlsanity.com/paper/2505.17332/) (2025)
- [Prompting Diverse Ideas: Increasing AI Idea Variance](https://mlsanity.com/paper/2402.01727/) (2024)
- [Prompting a Large Language Model to Generate Diverse Motivational Messages: A Comparison with Human-Written Messages](https://mlsanity.com/paper/2308.13479/) (2023)
- [FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding](https://mlsanity.com/paper/2505.17330/) (2025)
- [Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition](https://mlsanity.com/paper/2408.09215/) (2024)
- [Transformer-based Models of Text Normalization for Speech Applications](https://mlsanity.com/paper/2202.00153/) (2022)
- [Low-Resource Self-Supervised Learning with SSL-Enhanced TTS](https://mlsanity.com/paper/2309.17020/) (2023)
- [An Automated End-to-End Open-Source Software for High-Quality Text-to-Speech Dataset Generation](https://mlsanity.com/paper/2402.16380/) (2024)

## Built on by (most cited first, 5 of 5)

- [Aligning LLMs for Multilingual Consistency in Enterprise Applications](https://mlsanity.com/paper/2509.23659/) (2025)
- [Can LLMs Narrate Tabular Data? An Evaluation Framework for Natural Language Representations of Text-to-SQL System Outputs](https://mlsanity.com/paper/2510.23854/) (2025)
- [FlexDoc: Parameterized Sampling for Diverse Multilingual Synthetic Documents for Training Document Understanding Models](https://mlsanity.com/paper/2510.02133/) (2025)
- [PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications](https://mlsanity.com/paper/2509.23879/) (2025)
- [Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models](https://mlsanity.com/paper/2603.25750/) (2026)

## Abstract (arXiv)

High-quality Text-to-Speech (TTS) model training requires extensive and diverse text and speech data. It is challenging to procure such data from real sources due to issues of domain specificity, licensing, and scalability. Large language models (LLMs) can certainly generate textual data, but they create repetitive text with insufficient variation in the prompt during the generation process. Another important aspect in TTS training data is text normalization. Tools for normalization might occasionally introduce anomalies or overlook valuable patterns, and thus impact data quality. Furthermore, it is also impractical to rely on voice artists for large scale speech recording in commercial TTS systems with standardized voices. To address these challenges, we propose SpeechWeave, a synthetic speech data generation pipeline that is capable of automating the generation of multilingual, domain-specific datasets for training TTS models. Our experiments reveal that our pipeline generates data that is 10-48% more diverse than the baseline across various linguistic and phonetic metrics, along with speaker-standardized speech audio while generating approximately 97% correctly normalized text. Our approach enables scalable, high-quality data generation for TTS training, improving diversity, normalization, and voice consistency in the generated datasets.