# USEV: Universal Speaker Extraction with Visual Cue

arXiv: https://arxiv.org/abs/2109.14831 · HTML: https://mlsanity.com/paper/2109.14831/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2021.

## Built on by (most cited first, 11 of 11)

- [Speaker Extraction with Co-Speech Gestures Cue](https://mlsanity.com/paper/2203.16840/) (2022)
- [C2AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction](https://mlsanity.com/paper/2504.00750/) (2025)
- [Plug-and-Play Co-Occurring Face Attention for Robust Audio-Visual Speaker Extraction](https://mlsanity.com/paper/2505.20635/) (2025)
- [Late Audio-Visual Fusion for In-The-Wild Speaker Diarization](https://mlsanity.com/paper/2211.01299/) (2022)
- [SelectTSL: Prompt-Guided Selective Target Sound Localization in Complex Scenarios](https://mlsanity.com/paper/2607.02343/) (2026)
- [VCSE: Time-Domain Visual-Contextual Speaker Extraction Network](https://mlsanity.com/paper/2210.06177/) (2022)
- [Noise-Robust Target-Speaker Voice Activity Detection Through Self-Supervised Pretraining](https://mlsanity.com/paper/2501.03184/) (2025)
- [Audio-Visual Speech Enhancement In Complex Scenarios With Separation And Dereverberation Joint Modeling](https://mlsanity.com/paper/2510.26825/) (2025)
- [ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2511.06288/) (2025)
- [The Voiceprint Fallacy: Why Voices Are Not Unique Biometric Imprints](https://mlsanity.com/paper/2608.07980/) (2026)
- [Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extraction](https://mlsanity.com/paper/2609.30631/) (2026)

## Abstract (arXiv)

A speaker extraction algorithm seeks to extract the target speaker's speech from a multi-talker speech mixture. The prior studies focus mostly on speaker extraction from a highly overlapped multi-talker speech mixture. However, the target-interference speaker overlapping ratios could vary over a wide range from 0% to 100% in natural speech communication, furthermore, the target speaker could be absent in the speech mixture, the speech mixtures in such universal multi-talker scenarios are described as general speech mixtures. The speaker extraction algorithm requires an auxiliary reference, such as a video recording or a pre-recorded speech, to form top-down auditory attention on the target speaker. We advocate that a visual cue, i.e., lip movement, is more informative than an audio cue, i.e., pre-recorded speech, to serve as the auxiliary reference for speaker extraction in disentangling the target speaker from a general speech mixture. In this paper, we propose a universal speaker extraction network with a visual cue, that works for all multi-talker scenarios. In addition, we propose a scenario-aware differentiated loss function for network training, to balance the network performance over different target-interference speaker pairing scenarios. The experimental results show that our proposed method outperforms various competitive baselines for general speech mixtures in terms of signal fidelity.