mlsanityData to 14 Oct 2026

Real-Time Streaming Mel Vocoding with Generative Flow Matching

arXiv 2509.15085 ↗First posted 18 Sep 2025eess.AS cs.LG cs.SDcited by 1

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

The task of Mel vocoding, i.e., the inversion of a Mel magnitude spectrogram to an audio waveform, is still a key component in many text-to-speech (TTS) systems today. Based on generative flow matching, our prior work on generative STFT phase retrieval (DiffPhase), and the pseudoinverse operator of the Mel filterbank, we develop MelFlow, a streaming-capable generative Mel vocoder for speech sampled at 16 kHz with an algorithmic latency of only 32 ms and a total latency of 48 ms. We show real-time streaming capability at this latency not only in theory, but in practice on a consumer laptop GPU. Furthermore, we show that our model achieves substantially better PESQ and SI-SDR values compared to well-established not streaming-capable baselines for Mel vocoding including HiFi-GAN.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 27 of its 33 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

27 of 33 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Diffusion Buffer: Online Diffusion-based Speech Enhancement with Sub-Second Latency3 Jun
  2. 2025FlowDec: A flow-based full-band general audio codec with high perceptual quality3 Mar
  3. 2022DiffPhase: Generative Diffusion-based STFT Phase Retrieval8 Nov
  4. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  5. 2022Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain31 Mar
  6. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 21

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  4. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  6. 2018SDR - half-baked or well done?cited by 423
  7. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
  8. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  9. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 192
  10. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 153
  11. 2024EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberationcited by 64
  12. 2024Open-Source Conversational AI with SpeechBrain 1.0cited by 56
9 more
  1. 2022HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancementcited by 49
  2. 2020StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalizationcited by 26
  3. 2025Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessmentcited by 13
  4. 2021SubSpectral Normalization for Neural Audio Data Processingcited by 11
  5. 2022Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorchcited by 8
  6. 2024FreeV: Free Lunch For Vocoders Through Pseudo Inversed Mel Filtercited by 6
  7. 2021A Streamwise GAN Vocoder for Wideband Speech Coding at Very Low Bit Ratecited by 5
  8. 2023A Flexible Online Framework for Projection-Based STFT Phase Retrievalcited by 3
  9. 2022Framewise WaveGAN: High Speed Adversarial Vocoder in Time Domain with Very Low Computational Complexitycited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Real-Time Streamable Generative Speech Restoration with Flow Matchingcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026X-Pred MeanFlow for Streaming Token-to-Mel Speech Decoding ↗
  2. 2025WaveFM: A High-Fidelity and Efficient Vocoder Based on Flow Matching
  3. 2025UniverSR: Unified and Versatile Audio Super-Resolution via Vocoder-Free Flow Matching
  4. 2026P2Flow: Phoneme-aware Progressive Flow Matching for Extreme Speech Super-Resolution
  5. 2026WavFlow: Audio Generation in Waveform Space
  6. 2025Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decomposition
  7. 2025Towards Real-Time Generative Speech Restoration with Flow-Matching
  8. 2026Real-time Speech Restoration using Data Prediction Mean Flows