mlsanityData to 7 Oct 2026

RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement

arXiv 2407.07825 ↗First posted 10 Jul 2024cs.SD cs.CV cs.MMcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we aim to generate clean speech frame by frame from a live video stream and a noisy audio stream without relying on future inputs. To this end, we propose RT-LA-VocE, which completely re-designs every component of LA-VocE, a state-of-the-art non-causal audio-visual speech enhancement model, to perform causal real-time inference with a 40ms input frame. We do so by devising new visual and audio encoders that rely solely on past frames, replacing the Transformer encoder with the Emformer, and designing a new causal neural vocoder C-HiFi-GAN. On the popular AVSpeech dataset, we show that our algorithm achieves state-of-the-art results in all real-time scenarios. More importantly, each component is carefully tuned to minimize the algorithm latency to the theoretical minimum (40ms) while maintaining a low end-to-end processing latency of 28.15ms per frame, enabling real-time frame-by-frame enhancement with minimal delay.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 21 of its 37 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

21 of 37 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  3. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  4. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  5. 2020Real Time Speech Enhancement in the Waveform Domaincited by 183
  6. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  7. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Resultscited by 128
  8. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  9. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  10. 2019Music Source Separation in the Waveform Domaincited by 98
  11. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 76
  12. 2018End-to-end Audiovisual Speech Recognitioncited by 71
9 more
  1. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  2. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  3. 2020Muse: Multi-modal target speaker extraction with visual cuescited by 21
  4. 2022VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformercited by 14
  5. 2022Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillationcited by 8
  6. 2022LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoderscited by 8
  7. 2020UNetGAN: A Robust Speech Enhancement Approach in Time Domain for Extremely Low Signal-to-noise Ratio Conditioncited by 5
  8. 2023Real-Time Audio-Visual End-to-End Speech Enhancementcited by 5
  9. 2021Towards Robust Real-time Audio-Visual Speech Enhancementcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representationscited by 4
  2. 2025Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extractioncited by 2
  3. 2025Real-Time System for Audio-Visual Target Speech Enhancementcited by 0
  4. 2025BridgeVoC: Revitalizing Neural Vocoder from a Restoration Perspectivecited by 0