# MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues

arXiv: https://arxiv.org/abs/2412.08247 · HTML: https://mlsanity.com/paper/2412.08247/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 17 of 17)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [My lips are concealed: Audio-visual speech enhancement through obstructions](https://mlsanity.com/paper/1907.04975/) (2019)
- [FaceFilter: Audio-visual speech separation using still images](https://mlsanity.com/paper/2005.07074/) (2020)
- [Muse: Multi-modal target speaker extraction with visual cues](https://mlsanity.com/paper/2010.07775/) (2020)
- [Multimodal Attention Fusion for Target Speaker Extraction](https://mlsanity.com/paper/2102.01326/) (2021)
- [Robust Unsupervised Audio-visual Speech Enhancement Using a Mixture of Variational Autoencoders](https://mlsanity.com/paper/1911.03930/) (2019)
- [WeSep: A Scalable and Flexible Toolkit Towards Generalizable Target Speaker Extraction](https://mlsanity.com/paper/2409.15799/) (2024)
- [NeuroHeed: Neuro-Steered Speaker Extraction using EEG Signals](https://mlsanity.com/paper/2307.14303/) (2023)
- [Switching Variational Auto-Encoders for Noise-Agnostic Audio-visual Speech Enhancement](https://mlsanity.com/paper/2102.04144/) (2021)
- [ImagineNET: Target Speaker Extraction with Intermittent Visual Cue through Embedding Inpainting](https://mlsanity.com/paper/2211.00109/) (2022)
- [Iterative autoregression: a novel trick to improve your low-latency speech enhancement model](https://mlsanity.com/paper/2211.01751/) (2022)
- [Robust Speaker Extraction Network Based on Iterative Refined Adaptation](https://mlsanity.com/paper/2011.02102/) (2020)
- [Multi-Level Speaker Representation for Target Speaker Extraction](https://mlsanity.com/paper/2410.16059/) (2024)
- [Audio-Visual Active Speaker Extraction for Sparsely Overlapped Multi-talker Speech](https://mlsanity.com/paper/2309.08408/) (2023)

## Abstract (arXiv)

Audio-visual Target Speaker Extraction (AV-TSE) aims to isolate the speech of a specific target speaker from an audio mixture using time-synchronized visual cues. In real-world scenarios, visual cues are not always available due to various impairments, which undermines the stability of AV-TSE. Despite this challenge, humans can maintain attentional momentum over time, even when the target speaker is not visible. In this paper, we introduce the Momentum Multi-modal target Speaker Extraction (MoMuSE), which retains a speaker identity momentum in memory, enabling the model to continuously track the target speaker. Designed for real-time inference, MoMuSE extracts the current speech window with guidance from both visual cues and dynamically updated speaker momentum. Experimental results demonstrate that MoMuSE exhibits significant improvement, particularly in scenarios with severe impairment of visual cues.