# Plug-and-Play Co-Occurring Face Attention for Robust Audio-Visual Speaker Extraction

arXiv: https://arxiv.org/abs/2505.20635 · HTML: https://mlsanity.com/paper/2505.20635/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [Attention is All You Need in Speech Separation](https://mlsanity.com/paper/2010.13154/) (2020)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Wavesplit: End-to-End Speech Separation by Speaker Clustering](https://mlsanity.com/paper/2002.08933/) (2020)
- [TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separation](https://mlsanity.com/paper/2209.03952/) (2022)
- [VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition](https://mlsanity.com/paper/2009.04323/) (2020)
- [Time Domain Audio Visual Speech Separation](https://mlsanity.com/paper/1904.03760/) (2019)
- [SpEx: Multi-Scale Time Domain Speaker Extraction Network](https://mlsanity.com/paper/2004.08326/) (2020)
- [Deep attractor network for single-microphone speaker separation](https://mlsanity.com/paper/1611.08930/) (2016)
- [Muse: Multi-modal target speaker extraction with visual cues](https://mlsanity.com/paper/2010.07775/) (2020)
- [Selective Listening by Synchronizing Speech with Lips](https://mlsanity.com/paper/2106.07150/) (2021)
- [AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker Extraction](https://mlsanity.com/paper/2306.14170/) (2023)
- [USEV: Universal Speaker Extraction with Visual Cue](https://mlsanity.com/paper/2109.14831/) (2021)
- [Multimodal Attention Fusion for Target Speaker Extraction](https://mlsanity.com/paper/2102.01326/) (2021)
- [A Hybrid Continuity Loss to Reduce Over-Suppression for Time-domain Target Speaker Extraction](https://mlsanity.com/paper/2203.16843/) (2022)
- [NeuroHeed: Neuro-Steered Speaker Extraction using EEG Signals](https://mlsanity.com/paper/2307.14303/) (2023)
- [Target Confusion in End-to-end Speaker Extraction: Analysis and Approaches](https://mlsanity.com/paper/2204.01355/) (2022)
- [X-SepFormer: End-to-end Speaker Extraction Network with Explicit Optimization on Speaker Confusion](https://mlsanity.com/paper/2303.05023/) (2023)
- [NeuroHeed+: Improving Neuro-steered Speaker Extraction with Joint Auditory Attention Detection](https://mlsanity.com/paper/2312.07513/) (2023)
- [Speaker Extraction with Co-Speech Gestures Cue](https://mlsanity.com/paper/2203.16840/) (2022)
- [ImagineNET: Target Speaker Extraction with Intermittent Visual Cue through Embedding Inpainting](https://mlsanity.com/paper/2211.00109/) (2022)
- [Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction](https://mlsanity.com/paper/2310.19644/) (2023)
- [SA-SDR: A novel loss function for separation of meeting style data](https://mlsanity.com/paper/2110.15581/) (2021)

## Built on by (most cited first, 2 of 2)

- [From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios](https://mlsanity.com/paper/2608.08510/) (2026)
- [AVTR-1: Open Stack for Real-Time Interactive Avatars](https://mlsanity.com/paper/2609.22913/) (2026)

## Abstract (arXiv)

Audio-visual speaker extraction isolates a target speaker's speech from a mixture speech signal conditioned on a visual cue, typically using the target speaker's face recording. However, in real-world scenarios, other co-occurring faces are often present on-screen, providing valuable speaker activity cues in the scene. In this work, we introduce a plug-and-play inter-speaker attention module to process these flexible numbers of co-occurring faces, allowing for more accurate speaker extraction in complex multi-person environments. We integrate our module into two prominent models: the AV-DPRNN and the state-of-the-art AV-TFGridNet. Extensive experiments on diverse datasets, including the highly overlapped VoxCeleb2 and sparsely overlapped MISP, demonstrate that our approach consistently outperforms baselines. Furthermore, cross-dataset evaluations on LRS2 and LRS3 confirm the robustness and generalizability of our method.