mlsanityData to 14 Oct 2026

Real-Time System for Audio-Visual Target Speech Enhancement

arXiv 2509.20741 ↗First posted 25 Sep 2025eess.AS cs.ET cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a live demonstration for RAVEN, a real-time audio-visual speech enhancement system designed to run entirely on a CPU. In single-channel, audio-only settings, speech enhancement is traditionally approached as the task of extracting clean speech from environmental noise. More recent work has explored the use of visual cues, such as lip movements, to improve robustness, particularly in the presence of interfering speakers. However, to our knowledge, no prior work has demonstrated an interactive system for real-time audio-visual speech enhancement operating on CPU hardware. RAVEN fills this gap by using pretrained visual embeddings from an audio-visual speech recognition model to encode lip movement information. The system generalizes across environmental noise, interfering speakers, transient sounds, and even singing voices. In this demonstration, attendees will be able to experience live audio-visual target speech enhancement using a microphone and webcam setup, with clean speech playback through headphones.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 24 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

16 of 24 references.

References · 16

  1. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  2. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 162
  3. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  4. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 109
  5. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  6. 2017Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networkscited by 36
  7. 2019Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoderscited by 27
  8. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  9. 2024FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matchingcited by 23
  10. 2018Lip-Reading Driven Deep Learning Approach for Speech Enhancementcited by 14
  11. 2020Improved Lite Audio-Visual Speech Enhancementcited by 12
  12. 2021On The Compensation Between Magnitude and Phase in Speech Separationcited by 12
4 more
  1. 2022LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoderscited by 8
  2. 2023Real-Time Audio-Visual End-to-End Speech Enhancementcited by 5
  3. 2024RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancementcited by 4
  4. 2025Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representationscited by 4

Built on by

Not cited yet.