mlsanityData to 7 Oct 2026

Wave-U-Net Discriminator: Fast and Lightweight Discriminator for Generative Adversarial Network-Based Speech Synthesis

arXiv 2303.13909 ↗First posted 24 Mar 2023cs.SD cs.LG eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

In speech synthesis, a generative adversarial network (GAN), training a generator (speech synthesizer) and a discriminator in a min-max game, is widely used to improve speech quality. An ensemble of discriminators is commonly used in recent neural vocoders (e.g., HiFi-GAN) and end-to-end text-to-speech (TTS) systems (e.g., VITS) to scrutinize waveforms from multiple perspectives. Such discriminators allow synthesized speech to adequately approach real speech; however, they require an increase in the model size and computation time according to the increase in the number of discriminators. Alternatively, this study proposes a Wave-U-Net discriminator, which is a single but expressive discriminator with Wave-U-Net architecture. This discriminator is unique; it can assess a waveform in a sample-wise manner with the same resolution as the input signal, while extracting multilevel features via an encoder and decoder with skip connections. This architecture provides a generator with sufficiently rich information for the synthesized speech to be closely matched to the real speech. During the experiments, the proposed ideas were applied to a representative neural vocoder (HiFi-GAN) and an end-to-end TTS system (VITS). The results demonstrate that the proposed models can achieve comparable speech quality with a 2.31 times faster and 14.5 times more lightweight discriminator when used in HiFi-GAN and a 1.90 times faster and 9.62 times more lightweight discriminator when used in VITS. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/waveunetd/.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 35 of its 43 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

35 of 43 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2016Layer Normalizationcited by 5,579
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  8. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  10. 2016Least Squares Generative Adversarial Networkscited by 1,000
  11. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  12. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
23 more
  1. 2018Which Training Methods for GANs do actually Converge?cited by 672
  2. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  3. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  4. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  5. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  6. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  7. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  8. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  9. 2018Efficient Neural Audio Synthesiscited by 218
  10. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  11. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  12. 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 165
  13. 2020A U-Net Based Discriminator for Generative Adversarial Networkscited by 128
  14. 2018Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separationcited by 125
  15. 2020End-to-End Adversarial Text-to-Speechcited by 77
  16. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  17. 2020Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speechcited by 63
  18. 2017JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesiscited by 57
  19. 2022iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transformcited by 30
  20. 2020VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Networkcited by 26
  21. 2020StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalizationcited by 26
  22. 2020CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversioncited by 25
  23. 2021MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Framescited by 18

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2023iSTFTNet2: Faster and More Lightweight iSTFT-Based Neural Vocoder Using 1D-2D CNNcited by 3
  2. 2025Vocoder-Projected Feature Discriminatorcited by 2
  3. 2024Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representationscited by 0
  4. 2024Training Generative Adversarial Network-Based Vocoder with Limited Data Using Augmentation-Conditional Discriminatorcited by 0