mlsanityData to 14 Oct 2026

iSTFTNet2: Faster and More Lightweight iSTFT-Based Neural Vocoder Using 1D-2D CNN

arXiv 2308.07117 ↗First posted 14 Aug 2023cs.SD cs.LG eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

The inverse short-time Fourier transform network (iSTFTNet) has garnered attention owing to its fast, lightweight, and high-fidelity speech synthesis. It obtains these characteristics using a fast and lightweight 1D CNN as the backbone and replacing some neural processes with iSTFT. Owing to the difficulty of a 1D CNN to model high-dimensional spectrograms, the frequency dimension is reduced via temporal upsampling. However, this strategy compromises the potential to enhance the speed. Therefore, we propose iSTFTNet2, an improved variant of iSTFTNet with a 1D-2D CNN that employs 1D and 2D CNNs to model temporal and spectrogram structures, respectively. We designed a 2D CNN that performs frequency upsampling after conversion in a few-frequency space. This design facilitates the modeling of high-dimensional spectrograms without compromising the speed. The results demonstrated that iSTFTNet2 made iSTFTNet faster and more lightweight with comparable speech quality. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/istftnet2/.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 36 of its 44 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

36 of 44 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  7. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  8. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  9. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  11. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  12. 2016Least Squares Generative Adversarial Networkscited by 1,001
24 more
  1. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  2. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  3. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  4. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  5. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  6. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  7. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  8. 2018Efficient Neural Audio Synthesiscited by 219
  9. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 215
  10. 2022NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Qualitycited by 152
  11. 2020End-to-End Adversarial Text-to-Speechcited by 77
  12. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  13. 2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
  14. 2020Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speechcited by 63
  15. 2019CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversioncited by 46
  16. 2019DurIAN: Duration Informed Attention Network For Multimodal Synthesiscited by 33
  17. 2022iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transformcited by 32
  18. 2020VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Networkcited by 26
  19. 2020StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalizationcited by 26
  20. 2021Fre-GAN: Adversarial Frequency-consistent Audio Synthesiscited by 24
  21. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
  22. 2019Expediting TTS Synthesis with Adversarial Vocodingcited by 10
  23. 2018Generative adversarial network-based approach to signal reconstruction from magnitude spectrogramscited by 8
  24. 2023Wave-U-Net Discriminator: Fast and Lightweight Discriminator for Generative Adversarial Network-Based Speech Synthesiscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2025Vocoder-Projected Feature Discriminatorcited by 2
  2. 2024Training Generative Adversarial Network-Based Vocoder with Limited Data Using Augmentation-Conditional Discriminatorcited by 0
  3. 2026Toward Complex-Valued Neural Networks for Waveform Generationcited by 0
  4. 2026Paradee: Distilling Kokoro-82M into an 8M-Parameter Single-Voice Text-to-Speech Modelcited by 0