mlsanityData to 14 Oct 2026

FastFit: Towards Real-Time Iterative Neural Vocoder by Replacing U-Net Encoder With Multiple STFTs

arXiv 2305.10823 ↗First posted 18 May 2023eess.AS cs.LG cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper presents FastFit, a novel neural vocoder architecture that replaces the U-Net encoder with multiple short-time Fourier transforms (STFTs) to achieve faster generation rates without sacrificing sample quality. We replaced each encoder block with an STFT, with parameters equal to the temporal resolution of each decoder block, leading to the skip connection. FastFit reduces the number of parameters and the generation time of the model by almost half while maintaining high fidelity. Through objective and subjective evaluations, we demonstrated that the proposed model achieves nearly twice the generation speed of baseline iteration-based vocoders while maintaining high sound quality. We further showed that FastFit produces sound qualities similar to those of other baselines in text-to-speech evaluation scenarios, including multi-speaker and zero-shot text-to-speech.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 34 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

30 of 34 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  7. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  9. 2016Least Squares Generative Adversarial Networkscited by 1,001
  10. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  12. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
18 more
  1. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  2. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  3. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  4. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  5. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  6. 2021Direct speech-to-speech translation with discrete unitscited by 149
  7. 2022FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesiscited by 136
  8. 2018Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separationcited by 125
  9. 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 74
  10. 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
  11. 2022SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shapingcited by 37
  12. 2022iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transformcited by 32
  13. 2022InferGrad: Improving Diffusion Models for Vocoder by Considering Inference in Trainingcited by 28
  14. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
  15. 2021GANSpeech: Adversarial Training for High-Fidelity Multi-Speaker Speech Synthesiscited by 18
  16. 2020JDI-T: Jointly trained Duration Informed Transformer for Text-To-Speech without Explicit Alignmentcited by 13
  17. 2022Adapter-Based Extension of Multi-Speaker Text-to-Speech Model for New Speakerscited by 9
  18. 2022Autovocoder: Fast Waveform Generation from a Learned Speech Representation using Differentiable Digital Signal Processingcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generationcited by 12