mlsanityData to 14 Oct 2026

Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations

arXiv 2507.21448 ↗First posted 29 Jul 2025eess.AS cs.ET cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Speech enhancement in audio-only settings remains challenging, particularly in the presence of interfering speakers. This paper presents a simple yet effective real-time audio-visual speech enhancement (AVSE) system, RAVEN, which isolates and enhances the on-screen target speaker while suppressing interfering speakers and background noise. We investigate how visual embeddings learned from audio-visual speech recognition (AVSR) and active speaker detection (ASD) contribute to AVSE across different SNR conditions and numbers of interfering speakers. Our results show concatenating embeddings from AVSR and ASD models provides the greatest improvement in low-SNR, multi-speaker environments, while AVSR embeddings alone perform best in noise-only scenarios. In addition, we develop a real-time streaming system that operates on a computer CPU and we provide a video demonstration and code repository. To our knowledge, this is the first open-source implementation of a real-time AVSE system.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

26 of 36 references.

References · 26

  1. 2018Deep contextualized word representationscited by 2,290
  2. 2018SDR - half-baked or well done?cited by 423
  3. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  4. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  5. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 162
  6. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  7. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 113
  8. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 109
  9. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  10. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  11. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  12. 2021DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio based on Deep Filteringcited by 38
14 more
  1. 2021Few-shot Image Classification: Just Use a Library of Pre-trained Feature Extractors and a Simple Classifiercited by 32
  2. 2023LoCoNet: Long-Short Context Network for Active Speaker Detectioncited by 25
  3. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  4. 2024FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matchingcited by 23
  5. 2021Personalized Speech Enhancement: New Models and Comprehensive Evaluationcited by 21
  6. 2018Lip-Reading Driven Deep Learning Approach for Speech Enhancementcited by 14
  7. 2020Lite Audio-Visual Speech Enhancementcited by 12
  8. 2020Improved Lite Audio-Visual Speech Enhancementcited by 12
  9. 2021On The Compensation Between Magnitude and Phase in Speech Separationcited by 12
  10. 2018Exploring Tradeoffs in Models for Low-latency Speech Enhancementcited by 8
  11. 2023Real-Time Audio-Visual End-to-End Speech Enhancementcited by 5
  12. 2024RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancementcited by 4
  13. 2024ViSpeR: Multilingual Audio-Visual Speech Recognitioncited by 2
  14. 2024Music auto-tagging in the long tail: A few-shot approachcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025AV-Dialog: Spoken Dialogue Models with Audio-Visual Inputcited by 5
  2. 2025Real-Time System for Audio-Visual Target Speech Enhancementcited by 0
  3. 2026SG-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancementcited by 0
  4. 2026Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extractioncited by 0