# PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mapping

arXiv: https://arxiv.org/abs/2211.04610 · HTML: https://mlsanity.com/paper/2211.04610/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 8 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Differentiable Augmentation for Data-Efficient GAN Training](https://mlsanity.com/paper/2006.10738/) (2020)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling](https://mlsanity.com/paper/2104.02321/) (2021)
- [Chunked Autoregressive GAN for Conditional Waveform Synthesis](https://mlsanity.com/paper/2110.10139/) (2021)
- [NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates](https://mlsanity.com/paper/2206.08545/) (2022)
- [VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Network](https://mlsanity.com/paper/2007.15256/) (2020)
- [Fre-GAN: Adversarial Frequency-consistent Audio Synthesis](https://mlsanity.com/paper/2106.02297/) (2021)
- [Avocodo: Generative Adversarial Network for Artifact-free Vocoder](https://mlsanity.com/paper/2206.13404/) (2022)
- [TTS-by-TTS: TTS-driven Data Augmentation for Fast and High-Quality Speech Synthesis](https://mlsanity.com/paper/2010.13421/) (2020)

## Built on by (most cited first, 8 of 8)

- [PITS: Variational Pitch Inference without Fundamental Frequency for End-to-End Pitch-controllable TTS](https://mlsanity.com/paper/2302.12391/) (2023)
- [EDNet: A Versatile Speech Enhancement Framework with Gating Mamba Mechanism and Phase Shift-Invariant Training](https://mlsanity.com/paper/2506.16231/) (2025)
- [VIFS: An End-to-End Variational Inference for Foley Sound Synthesis](https://mlsanity.com/paper/2306.05004/) (2023)
- [Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codec](https://mlsanity.com/paper/2603.05887/) (2026)
- [PhasePerturbation: Speech Data Augmentation via Phase Perturbation for Automatic Speech Recognition](https://mlsanity.com/paper/2312.08571/) (2023)
- [MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidances](https://mlsanity.com/paper/2509.17143/) (2025)
- [Training Generative Adversarial Network-Based Vocoder with Limited Data Using Augmentation-Conditional Discriminator](https://mlsanity.com/paper/2403.16464/) (2024)
- [JenGAN: Stacked Shifted Filters in GAN-Based Speech Synthesis](https://mlsanity.com/paper/2406.06111/) (2024)

## Abstract (arXiv)

Previous generative adversarial network (GAN)-based neural vocoders are trained to reconstruct the exact ground truth waveform from the paired mel-spectrogram and do not consider the one-to-many relationship of speech synthesis. This conventional training causes overfitting for both the discriminators and the generator, leading to the periodicity artifacts in the generated audio signal. In this work, we present PhaseAug, the first differentiable augmentation for speech synthesis that rotates the phase of each frequency bin to simulate one-to-many mapping. With our proposed method, we outperform baselines without any architecture modification. Code and audio samples will be available at https://github.com/mindslab-ai/phaseaug.