mlsanityData to 7 Oct 2026

FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching

arXiv 2406.09286 ↗First posted 13 Jun 2024eess.AS cs.SDcited by 23

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

This work proposes an efficient method to enhance the quality of corrupted speech signals by leveraging both acoustic and visual cues. While existing diffusion-based approaches have demonstrated remarkable quality, their applicability is limited by slow inference speeds and computational complexity. To address this issue, we present FlowAVSE which enhances the inference speed and reduces the number of learnable parameters without degrading the output quality. In particular, we employ a conditional flow matching algorithm that enables the generation of high-quality speech in a single sampling step. Moreover, we increase efficiency by optimizing the underlying U-net architecture of diffusion-based systems. Our experiments demonstrate that FlowAVSE achieves 22 times faster inference speed and reduces the model size by half while maintaining the output quality. The demo page is available at: https://cyongong.github.io/FlowAVSE.github.io/

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 38 of its 44 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

38 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Seeing Through the Conversation: Audio-Visual Speech Separation based on Diffusion Model30 Oct
  2. 2023Audio-Visual Speech Enhancement with Score-Based Generative Models2 Jun
  3. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation22 Dec
  4. 2022Diffusion-based Generative Speech Source Separation31 Oct
  5. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  6. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May

Other references · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Flow Matching for Generative Modelingcited by 5,426
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2013Auto-Encoding Variational Bayescited by 3,980
  12. 2018Neural Ordinary Differential Equationscited by 3,733
20 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022Video Diffusion Modelscited by 2,280
  3. 2015Variational Inference with Normalizing Flowscited by 2,161
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  6. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  7. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 835
  8. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  9. 2018SDR - half-baked or well done?cited by 420
  10. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  11. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  12. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  13. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  14. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 110
  15. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  16. 2019Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoderscited by 27
  17. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  18. 2023TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learningcited by 15
  19. 2022Audio-Visual Speech Enhancement and Separation by Utilizing Multi-Modal Self-Supervised Embeddingscited by 7
  20. 2023Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 9.

23 papers.

Sort:

By area:

  1. 2025FlowSE: Flow Matching-based Speech Enhancementcited by 20
  2. 2025FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matchingcited by 17
  3. 2025FlowDec: A flow-based full-band general audio codec with high perceptual qualitycited by 15
  4. 2024Diffusion-based Unsupervised Audio-visual Speech Enhancementcited by 8
  5. 2024SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Modelcited by 6
  6. 2025BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Modelscited by 5
  7. 2026FlowSE-GRPO: Training Flow Matching Speech Enhancement via Online Reinforcement Learningcited by 5
  8. 2025Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representationscited by 4
  9. 2025Flowing Straighter with Conditional Flow Matching for Accurate Speech Enhancementcited by 3
  10. 2025Towards Real-Time Generative Speech Restoration with Flow-Matchingcited by 3
  11. 2025TimeFlow: Towards Stochastic-Aware and Efficient Time Series Generation via Flow Matching Modelingcited by 2
  12. 2025Diffusion-Based Unsupervised Audio-Visual Speech Separation in Noisy Environments with Noise Priorcited by 1
11 more
  1. 2026CodecFlow: Efficient Bandwidth Extension via Conditional Flow Matching in Neural Codec Latent Spacecited by 1
  2. 2025InfiniteAudio: Infinite-Length Audio Generation with Consistencycited by 0
  3. 2025Speech Enhancement based on cascaded two flowscited by 0
  4. 2025Real-Time System for Audio-Visual Target Speech Enhancementcited by 0
  5. 2026SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Samplingcited by 0
  6. 2026Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matchingcited by 0
  7. 2026DiT-Flow: Speech Enhancement Robust to Multiple Distortions based on Flow Matching in Latent Space and Diffusion Transformerscited by 0
  8. 2026Audio-visual Contrastive Alignment for Diffusion-based Visual-conditioned Speech Enhancementcited by 0
  9. 2026Beyond U-Net: A Latent-Representation-Aligned Skip-Free Backbone for Flow-Matching Speech Enhancementcited by 0
  10. 2026Discriminative Flow Matching: Beyond Time-Conditioning in Generative Restoration via Flow-State Representationscited by 0
  11. 2026SG-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Shortcut Flow Matching for Speech Enhancement: Step-Invariant flows via single stage training
  2. 2025Investigating training objective for flow matching-based speech enhancement
  3. 2025Compose Yourself: Average-Velocity Flow Matching for One-Step Speech Enhancement
  4. 2023AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement
  5. 2025Adaptive Deterministic Flow Matching for Target Speaker Extraction
  6. 2025MeanSE: Efficient Generative Speech Enhancement with Mean Flows
  7. 2026Geneses: Unified Generative Speech Enhancement and Separation
  8. 2026Rethinking Flow and Diffusion Bridge Models for Speech Enhancement