mlsanityData to 7 Oct 2026

Audio-Visual Speech Enhancement with Score-Based Generative Models

arXiv 2306.01432 ↗First posted 2 Jun 2023eess.AS cs.LGcited by 12

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

This paper introduces an audio-visual speech enhancement system that leverages score-based generative models, also known as diffusion models, conditioned on visual information. In particular, we exploit audio-visual embeddings obtained from a self-super-vised learning model that has been fine-tuned on lipreading. The layer-wise features of its transformer-based encoder are aggregated, time-aligned, and incorporated into the noise conditional score network. Experimental evaluations show that the proposed audio-visual speech enhancement system yields improved speech quality and reduces generative artifacts such as phonetic confusions with respect to the audio-only equivalent. The latter is supported by the word error rate of a downstream automatic speech recognition model, which decreases noticeably, especially at low input signal-to-noise ratios.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 27 of its 37 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

27 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  2. 2022Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain31 Mar

Other references · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  9. 2018SDR - half-baked or well done?cited by 420
  10. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  11. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  12. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
13 more
  1. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  2. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  3. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  4. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  5. 2020Lipreading using Temporal Convolutional Networkscited by 98
  6. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  7. 2020Deep Audio-Visual Learning: A Surveycited by 68
  8. 2023A Review of Deep Learning Techniques for Speech Processingcited by 52
  9. 2017Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networkscited by 36
  10. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  11. 2019Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoderscited by 27
  12. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  13. 2022LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoderscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 1; 2026: 4.

12 papers.

  1. 2024Diffusion Models for Audio Restorationcited by 32
  2. 2024FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matchingcited by 23
  3. 2024Diffusion-based Unsupervised Audio-visual Speech Enhancementcited by 8
  4. 2024Toward Enhancing Vehicle Color Recognition in Adverse Conditions: A Dataset and Benchmarkcited by 6
  5. 2024SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Modelcited by 6
  6. 2023AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancementcited by 5
  7. 2025LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Modelscited by 2
  8. 2026Robust Face Super-Resolution and Recognition Through Multi-Feature Aggregation in Diffusion Modelscited by 1
  9. 2024Multi-Feature Aggregation in Diffusion Models for Enhanced Face Super-Resolutioncited by 0
  10. 2026Bone-conduction Guided Multimodal Speech Enhancement with Conditional Diffusion Modelscited by 0
  11. 2026SAS: Semantic-aware Sampling for Generative Dataset Distillationcited by 0
  12. 2026SG-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Unsupervised speech enhancement with diffusion-based generative models
  2. 2026Audio-visual Contrastive Alignment for Diffusion-based Visual-conditioned Speech Enhancement
  3. 2024Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement
  4. 2023Diffusion-based speech enhancement with a weighted generative-supervised learning loss
  5. 2024Diffusion-based Generative Modeling with Discriminative Guidance for Streamable Speech Enhancement
  6. 2022Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration
  7. 2023Diffusion-Based Speech Enhancement in Matched and Mismatched Conditions Using a Heun-Based Sampler
  8. 2023Single and Few-step Diffusion for Generative Speech Enhancement