# Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancement

arXiv: https://arxiv.org/abs/2501.13372 · HTML: https://mlsanity.com/paper/2501.13372/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model](https://mlsanity.com/paper/2406.04904/) (2024)
- [MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement](https://mlsanity.com/paper/2104.03538/) (2021)
- [The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech](https://mlsanity.com/paper/2409.09305/) (2024)
- [Better speech synthesis through scaling](https://mlsanity.com/paper/2305.07243/) (2023)
- [MANNER: Multi-view Attention Network for Noise Erasure](https://mlsanity.com/paper/2203.02181/) (2022)
- [The Singing Voice Conversion Challenge 2023](https://mlsanity.com/paper/2306.14422/) (2023)
- [The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction](https://mlsanity.com/paper/2409.07001/) (2024)
- [Text is All You Need: Personalizing ASR Models using Controllable Speech Synthesis](https://mlsanity.com/paper/2303.14885/) (2023)
- [Improving Few-Shot Learning for Talking Face System with TTS Data Augmentation](https://mlsanity.com/paper/2303.05322/) (2023)
- [Efficient Personalized Speech Enhancement through Self-Supervised Learning](https://mlsanity.com/paper/2104.02017/) (2021)
- [Deep MOS Predictor for Synthetic Speech Using Cluster-Based Modeling](https://mlsanity.com/paper/2008.03710/) (2020)
- [Latent Filling: Latent Space Data Augmentation for Zero-shot Speech Synthesis](https://mlsanity.com/paper/2310.03538/) (2023)
- [Utilizing TTS Synthesized Data for Efficient Development of Keyword Spotting Model](https://mlsanity.com/paper/2407.18879/) (2024)
- [The Potential of Neural Speech Synthesis-based Data Augmentation for Personalized Speech Enhancement](https://mlsanity.com/paper/2211.07493/) (2022)
- [Zero Shot Text to Speech Augmentation for Automatic Speech Recognition on Low-Resource Accented Speech Corpora](https://mlsanity.com/paper/2409.11107/) (2024)

## Built on by (most cited first, 3 of 3)

- [Analyzing and Improving Speaker Similarity Assessment for Speech Synthesis](https://mlsanity.com/paper/2507.02176/) (2025)
- [Semantics-Aware Generative Latent Data Augmentation for Learning in Low-Resource Domains](https://mlsanity.com/paper/2602.02841/) (2026)
- [ZeSTA: Zero-Shot TTS Augmentation with Domain-Conditioned Training for Data-Efficient Personalized Speech Synthesis](https://mlsanity.com/paper/2603.04219/) (2026)

## Abstract (arXiv)

This paper presents a new challenge that calls for zero-shot text-to-speech (TTS) systems to augment speech data for the downstream task, personalized speech enhancement (PSE), as part of the Generative Data Augmentation workshop at ICASSP 2025. Collecting high-quality personalized data is challenging due to privacy concerns and technical difficulties in recording audio from the test scene. To address these issues, synthetic data generation using generative models has gained significant attention. In this challenge, participants are tasked first with building zero-shot TTS systems to augment personalized data. Subsequently, PSE systems are asked to be trained with this augmented personalized dataset. Through this challenge, we aim to investigate how the quality of augmented data generated by zero-shot TTS models affects PSE model performance. We also provide baseline experiments using open-source zero-shot TTS models to encourage participation and benchmark advancements. Our baseline code implementation and checkpoints are available online.