mlsanityData to 7 Oct 2026

iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform

arXiv 2203.02395 ↗First posted 4 Mar 2022cs.SD cs.LG eess.AScited by 30

Not on the map. Why?

Abstract (arXiv)

In recent text-to-speech synthesis and voice conversion systems, a mel-spectrogram is commonly applied as an intermediate representation, and the necessity for a mel-spectrogram vocoder is increasing. A mel-spectrogram vocoder must solve three inverse problems: recovery of the original-scale magnitude spectrogram, phase reconstruction, and frequency-to-time conversion. A typical convolutional mel-spectrogram vocoder solves these problems jointly and implicitly using a convolutional neural network, including temporal upsampling layers, when directly calculating a raw waveform. Such an approach allows skipping redundant processes during waveform synthesis (e.g., the direct reconstruction of high-dimensional original-scale spectrograms). By contrast, the approach solves all problems in a black box and cannot effectively employ the time-frequency structures existing in a mel-spectrogram. We thus propose iSTFTNet, which replaces some output-side layers of the mel-spectrogram vocoder with the inverse short-time Fourier transform (iSTFT) after sufficiently reducing the frequency dimension using upsampling layers, reducing the computational cost from black-box modeling and avoiding redundant estimations of high-dimensional spectrograms. During our experiments, we applied our ideas to three HiFi-GAN variants and made the models faster and more lightweight with a reasonable speech quality. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/istftnet/.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 34 of its 40 references that are in the map. First posted on arXiv 4 Mar 2022.

Builds on

34 of 40 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  8. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  9. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  10. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  11. 2016Least Squares Generative Adversarial Networkscited by 1,000
  12. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
22 more
  1. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  2. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  3. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  4. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  5. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  6. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  7. 2018Efficient Neural Audio Synthesiscited by 218
  8. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 214
  9. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  10. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  11. 2019High Fidelity Speech Synthesis with Adversarial Networkscited by 90
  12. 2018ClariNet: Parallel Wave Generation in End-to-End Text-to-Speechcited by 70
  13. 2020Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speechcited by 63
  14. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  15. 2020A Spectral Energy Distance for Parallel Speech Synthesiscited by 42
  16. 2019DurIAN: Duration Informed Attention Network For Multimodal Synthesiscited by 33
  17. 2020VocGAN: A High-Fidelity Real-time Vocoder with a Hierarchically-nested Adversarial Networkcited by 26
  18. 2020StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalizationcited by 26
  19. 2020CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversioncited by 25
  20. 2021MaskCycleGAN-VC: Learning Non-parallel Voice Conversion with Filling in Framescited by 18
  21. 2019Expediting TTS Synthesis with Adversarial Vocodingcited by 10
  22. 2018Generative adversarial network-based approach to signal reconstruction from magnitude spectrogramscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 6; 2025: 9; 2026: 4.

30 papers.

Sort:

By area:

  1. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 150
  2. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  3. 2022Musika! Fast Infinite Waveform Music Generationcited by 24
  4. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
  5. 2023EmoSpeech: Guiding FastSpeech2 Towards Emotional Text to Speechcited by 14
  6. 2024PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generationcited by 12
  7. 2024FreGrad: Lightweight and Fast Frequency-aware Diffusion Vocodercited by 11
  8. 2023HiFTNet: A Fast High-Quality Neural Vocoder with Harmonic-plus-Noise Filter and Inverse Short Time Fourier Transformcited by 10
  9. 2024RFWave: Multi-band Rectified Flow for Audio Waveform Reconstructioncited by 9
  10. 2023Msanii: High Fidelity Music Synthesis on a Shoestring Budgetcited by 6
  11. 2025SafeSpeech: Robust and Universal Voice Protection Against Malicious Speech Synthesiscited by 5
  12. 2022A Deep Reinforcement Learning Blind AI in DareFightingICEcited by 4
18 more
  1. 2023Wave-U-Net Discriminator: Fast and Lightweight Discriminator for Generative Adversarial Network-Based Speech Synthesiscited by 4
  2. 2024UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Promptscited by 4
  3. 2023iSTFTNet2: Faster and More Lightweight iSTFT-Based Neural Vocoder Using 1D-2D CNNcited by 3
  4. 2025Next Tokens Denoising for Speech Synthesiscited by 3
  5. 2024Lightweight Zero-shot Text-to-Speech with Mixture of Adapterscited by 2
  6. 2025Vocoder-Projected Feature Discriminatorcited by 2
  7. 2023FastFit: Towards Real-Time Iterative Neural Vocoder by Replacing U-Net Encoder With Multiple STFTscited by 1
  8. 2025Designing Neural Synthesizers for Low-Latency Interactioncited by 1
  9. 2025Towards Pretraining Robust ASR Foundation Model with Acoustic-Aware Data Augmentationcited by 1
  10. 2024Training Generative Adversarial Network-Based Vocoder with Limited Data Using Augmentation-Conditional Discriminatorcited by 0
  11. 2025RingFormer: A Neural Vocoder with Ring Attention and Convolution-Augmented Transformercited by 0
  12. 2025Spiking Vocos: An Energy-Efficient Neural Vocodercited by 0
  13. 2025E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesiscited by 0
  14. 2025BridgeVoC: Revitalizing Neural Vocoder from a Restoration Perspectivecited by 0
  15. 2026Toward Complex-Valued Neural Networks for Waveform Generationcited by 0
  16. 2026WaveNeXt 2: ConvNeXt-Based Fast Neural Vocoders With Residual Denoising and Sub-Modeling for GAN and Diffusion Modelscited by 0
  17. 2026NAC: Neural Action Codec for Vision-Language-Action Modelscited by 0
  18. 2026Adaptive Oscillatory Inductive Bias for Modeling Sharp Prosodic Dynamics in Diffusion-Based TTScited by 0