# Real-Time Streaming Mel Vocoding with Generative Flow Matching

arXiv: https://arxiv.org/abs/2509.15085 · HTML: https://mlsanity.com/paper/2509.15085/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 27 of 27)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2406.06185/) (2024)
- [Open-Source Conversational AI with SpeechBrain 1.0](https://mlsanity.com/paper/2407.00463/) (2024)
- [HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancement](https://mlsanity.com/paper/2203.13086/) (2022)
- [StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalization](https://mlsanity.com/paper/2011.01557/) (2020)
- [FlowDec: A flow-based full-band general audio codec with high perceptual quality](https://mlsanity.com/paper/2503.01485/) (2025)
- [Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment](https://mlsanity.com/paper/2502.05356/) (2025)
- [SubSpectral Normalization for Neural Audio Data Processing](https://mlsanity.com/paper/2103.13620/) (2021)
- [Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch](https://mlsanity.com/paper/2204.03418/) (2022)
- [DiffPhase: Generative Diffusion-based STFT Phase Retrieval](https://mlsanity.com/paper/2211.04332/) (2022)
- [FreeV: Free Lunch For Vocoders Through Pseudo Inversed Mel Filter](https://mlsanity.com/paper/2406.08196/) (2024)
- [Diffusion Buffer: Online Diffusion-based Speech Enhancement with Sub-Second Latency](https://mlsanity.com/paper/2506.02908/) (2025)
- [A Streamwise GAN Vocoder for Wideband Speech Coding at Very Low Bit Rate](https://mlsanity.com/paper/2108.04051/) (2021)
- [A Flexible Online Framework for Projection-Based STFT Phase Retrieval](https://mlsanity.com/paper/2309.07043/) (2023)
- [Framewise WaveGAN: High Speed Adversarial Vocoder in Time Domain with Very Low Computational Complexity](https://mlsanity.com/paper/2212.04532/) (2022)

## Built on by (most cited first, 1 of 1)

- [Real-Time Streamable Generative Speech Restoration with Flow Matching](https://mlsanity.com/paper/2512.19442/) (2025)

## Abstract (arXiv)

The task of Mel vocoding, i.e., the inversion of a Mel magnitude spectrogram to an audio waveform, is still a key component in many text-to-speech (TTS) systems today. Based on generative flow matching, our prior work on generative STFT phase retrieval (DiffPhase), and the pseudoinverse operator of the Mel filterbank, we develop MelFlow, a streaming-capable generative Mel vocoder for speech sampled at 16 kHz with an algorithmic latency of only 32 ms and a total latency of 48 ms. We show real-time streaming capability at this latency not only in theory, but in practice on a consumer laptop GPU. Furthermore, we show that our model achieves substantially better PESQ and SI-SDR values compared to well-established not streaming-capable baselines for Mel vocoding including HiFi-GAN.