mlsanityData to 14 Oct 2026

Look&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement

arXiv 2203.02216 ↗First posted 4 Mar 2022cs.SD cs.AI cs.MMcited by 20

Not on the map. Why?

Abstract (arXiv)

Active speaker detection and speech enhancement have become two increasingly attractive topics in audio-visual scenario understanding. According to their respective characteristics, the scheme of independently designed architecture has been widely used in correspondence to each single task. This may lead to the representation learned by the model being task-specific, and inevitably result in the lack of generalization ability of the feature based on multi-modal modeling. More recent studies have shown that establishing cross-modal relationship between auditory and visual stream is a promising solution for the challenge of audio-visual multi-task learning. Therefore, as a motivation to bridge the multi-modal associations in audio-visual tasks, a unified framework is proposed to achieve target speaker detection and speech enhancement with joint learning of audio-visual modeling in this study.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 37 of its 80 references that are in the map. First posted on arXiv 4 Mar 2022.

Builds on

37 of 80 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Layer Normalizationcited by 5,669
  4. 2017Squeeze-and-Excitation Networkscited by 4,789
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  7. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  8. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  9. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
  10. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  11. 2018Deep Audio-Visual Speech Recognitioncited by 322
  12. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
25 more
  1. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  2. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  3. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 212
  4. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  5. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  6. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  7. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 113
  8. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 109
  9. 2021One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learningcited by 87
  10. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 83
  11. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  12. 2021CTNet: Context-based Tandem Network for Semantic Segmentationcited by 43
  13. 2020Active Speakers in Contextcited by 41
  14. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 36
  15. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  16. 2021Class-aware Sounding Objects Localization via Audiovisual Correspondencecited by 32
  17. 2021How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wildcited by 31
  18. 2021UniCon: Unified Context Network for Robust Active Speaker Detectioncited by 28
  19. 2018On Learning Associations of Faces and Voicescited by 26
  20. 2019Naver at ActivityNet Challenge 2019 -- Task B Active Speaker Detection (AVA)cited by 26
  21. 2016Cross-modal Supervision for Learning Active Speaker Detection in Videocited by 21
  22. 2020Re-synchronization using the Hand Preceding Model for Multi-modal Fusion in Automatic Continuous Cued Speech Recognitioncited by 17
  23. 2018Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environmentscited by 12
  24. 2018End-to-end Audiovisual Speech Activity Detection with Bimodal Recurrent Neural Modelscited by 5
  25. 2021Decompose the Sounds and Pixels, Recompose the Eventscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 7; 2026: 4.

20 papers.

Sort:

By area:

  1. 2023A Light Weight Model for Active Speaker Detectioncited by 40
  2. 2023LoCoNet: Long-Short Context Network for Active Speaker Detectioncited by 25
  3. 2023Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipationcited by 20
  4. 2023Egocentric Auditory Attention Localization in Conversationscited by 18
  5. 2025AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentationcited by 16
  6. 2023CASP-Net: Rethinking Video Saliency Prediction from an Audio-VisualConsistency Perceptual Perspectivecited by 15
  7. 2025LASER: Lip Landmark Assisted Speaker Detection for Robustnesscited by 6
  8. 2025Revisiting Active Speaker Detection: An In-the-Wild Benchmark for Generalization and Robustnesscited by 6
  9. 2025Complementary and Contrastive Learning for Audio-Visual Segmentationcited by 5
  10. 2025Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancementcited by 3
  11. 2024Robust Active Speaker Detection in Noisy Environmentscited by 2
  12. 2024CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrievalcited by 2
8 more
  1. 2025GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detectioncited by 1
  2. 2024CLIP-VAD: Exploiting Vision-Language Models for Voice Activity Detectioncited by 0
  3. 2024FabuLight-ASD: Unveiling Speech Activity via Body Languagecited by 0
  4. 2025ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extractioncited by 0
  5. 2026Visual-Informed Speech Enhancement Using Attention-Based Beamformingcited by 0
  6. 2026EgoAdapt: Enhancing Robustness in Egocentric Interactive Speaker Detection Under Missing Modalitiescited by 0
  7. 2026C3ASD: Multi-Level Consistency-Driven Representation Learning for Robust Active Speaker Detectioncited by 0
  8. 2026Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videoscited by 0