mlsanityData to 7 Oct 2026

The Conversation: Deep Audio-Visual Speech Enhancement

arXiv 1804.04121 ↗First posted 11 Apr 2018cs.CV cs.SDcited by 107

Not on the map. Why?

Abstract (arXiv)

Our goal is to isolate individual speakers from multi-talker simultaneous speech in videos. Existing works in this area have focussed on trying to separate utterances from known speakers in controlled environments. In this paper, we propose a deep audio-visual speech enhancement network that is able to separate a speaker's voice given lip regions in the corresponding video, by predicting both the magnitude and the phase of the target signal. The method is applicable to speakers unheard and unseen during training, and for unconstrained environments. We demonstrate strong quantitative and qualitative results, isolating extremely challenging real-world examples.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 37; 2023: 19; 2024: 18; 2025: 10; 2026: 5.

107 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  3. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  4. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  5. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  6. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  7. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  8. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  9. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  10. 2022Visual Acoustic Matchingcited by 45
  11. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  12. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
38 more
  1. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  2. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  3. 2023Novel-View Acoustic Synthesiscited by 33
  4. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  5. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 30
  6. 2022Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detectioncited by 29
  7. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  8. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 27
  9. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 27
  10. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  11. 2022AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separationcited by 24
  12. 2024FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matchingcited by 23
  13. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  14. 2021Look Who's Talking: Active Speaker Detection in the Wildcited by 21
  15. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  16. 2022Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learningcited by 21
  17. 2022Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancementcited by 20
  18. 2022Sound Localization by Self-Supervised Time Delay Estimationcited by 18
  19. 2023Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parsercited by 18
  20. 2022Active Audio-Visual Separation of Dynamic Sound Sourcescited by 16
  21. 2023Target Speech Extraction with Conditional Diffusion Modelcited by 15
  22. 2023AdVerb: Visually Guided Audio Dereverberationcited by 15
  23. 2023TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learningcited by 15
  24. 2022VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformercited by 14
  25. 2021VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Overcited by 13
  26. 2021A cappella: Audio-visual Singing Voice Separationcited by 12
  27. 2023Audio-Visual Speech Enhancement with Score-Based Generative Modelscited by 12
  28. 2023IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separationcited by 11
  29. 2024Look Once to Hear: Target Speech Hearing with Noisy Examplescited by 11
  30. 2021Multimodal Attention Fusion for Target Speaker Extractioncited by 10
  31. 2022An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuitscited by 9
  32. 2023Speaker Recognition in Realistic Scenario Using Multimodal Datacited by 9
  33. 2023Seeing Through the Conversation: Audio-Visual Speech Separation based on Diffusion Modelcited by 9
  34. 2024FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wildcited by 9
  35. 2024The VoxCeleb Speaker Recognition Challenge: A Retrospectivecited by 9
  36. 2022ConceptBeam: Concept Driven Target Speech Extractioncited by 8
  37. 2022Lip-to-Speech Synthesis for Arbitrary Speakers in the Wildcited by 8
  38. 2022LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoderscited by 8

50 most cited of 107 All 107 (JSON)