# FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching

arXiv: https://arxiv.org/abs/2406.09286 · HTML: https://mlsanity.com/paper/2406.09286/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 38 of its 44 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Improving and generalizing flow-based generative models with minibatch optimal transport](https://mlsanity.com/paper/2302.00482/) (2023)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)

## Built on by (most cited first, 23 of 23)

- [FlowSE: Flow Matching-based Speech Enhancement](https://mlsanity.com/paper/2508.06840/) (2025)
- [FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching](https://mlsanity.com/paper/2505.19476/) (2025)
- [FlowDec: A flow-based full-band general audio codec with high perceptual quality](https://mlsanity.com/paper/2503.01485/) (2025)
- [Diffusion-based Unsupervised Audio-visual Speech Enhancement](https://mlsanity.com/paper/2410.05301/) (2024)
- [SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model](https://mlsanity.com/paper/2411.07751/) (2024)
- [BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models](https://mlsanity.com/paper/2505.22865/) (2025)
- [FlowSE-GRPO: Training Flow Matching Speech Enhancement via Online Reinforcement Learning](https://mlsanity.com/paper/2601.16483/) (2026)
- [Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations](https://mlsanity.com/paper/2507.21448/) (2025)
- [Towards Real-Time Generative Speech Restoration with Flow-Matching](https://mlsanity.com/paper/2510.16997/) (2025)
- [Flowing Straighter with Conditional Flow Matching for Accurate Speech Enhancement](https://mlsanity.com/paper/2508.20584/) (2025)
- [TimeFlow: Towards Stochastic-Aware and Efficient Time Series Generation via Flow Matching Modeling](https://mlsanity.com/paper/2511.07968/) (2025)
- [Diffusion-Based Unsupervised Audio-Visual Speech Separation in Noisy Environments with Noise Prior](https://mlsanity.com/paper/2509.14379/) (2025)
- [CodecFlow: Efficient Bandwidth Extension via Conditional Flow Matching in Neural Codec Latent Space](https://mlsanity.com/paper/2603.02022/) (2026)
- [InfiniteAudio: Infinite-Length Audio Generation with Consistency](https://mlsanity.com/paper/2506.03020/) (2025)
- [Speech Enhancement based on cascaded two flows](https://mlsanity.com/paper/2508.06842/) (2025)
- [Real-Time System for Audio-Visual Target Speech Enhancement](https://mlsanity.com/paper/2509.20741/) (2025)
- [SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling](https://mlsanity.com/paper/2602.01394/) (2026)
- [Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching](https://mlsanity.com/paper/2603.12837/) (2026)
- [DiT-Flow: Speech Enhancement Robust to Multiple Distortions based on Flow Matching in Latent Space and Diffusion Transformers](https://mlsanity.com/paper/2603.21608/) (2026)
- [Audio-visual Contrastive Alignment for Diffusion-based Visual-conditioned Speech Enhancement](https://mlsanity.com/paper/2606.23712/) (2026)
- [Beyond U-Net: A Latent-Representation-Aligned Skip-Free Backbone for Flow-Matching Speech Enhancement](https://mlsanity.com/paper/2606.24745/) (2026)
- [Discriminative Flow Matching: Beyond Time-Conditioning in Generative Restoration via Flow-State Representations](https://mlsanity.com/paper/2609.04525/) (2026)
- [SG-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2609.18009/) (2026)

## Abstract (arXiv)

This work proposes an efficient method to enhance the quality of corrupted speech signals by leveraging both acoustic and visual cues. While existing diffusion-based approaches have demonstrated remarkable quality, their applicability is limited by slow inference speeds and computational complexity. To address this issue, we present FlowAVSE which enhances the inference speed and reduces the number of learnable parameters without degrading the output quality. In particular, we employ a conditional flow matching algorithm that enables the generation of high-quality speech in a single sampling step. Moreover, we increase efficiency by optimizing the underlying U-net architecture of diffusion-based systems. Our experiments demonstrate that FlowAVSE achieves 22 times faster inference speed and reduces the model size by half while maintaining the output quality. The demo page is available at: https://cyongong.github.io/FlowAVSE.github.io/