# Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extraction

arXiv: https://arxiv.org/abs/2609.30631 · HTML: https://mlsanity.com/paper/2609.30631/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation](https://mlsanity.com/paper/1804.03619/) (2018)
- [VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking](https://mlsanity.com/paper/1810.04826/) (2018)
- [ICASSP 2023 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2303.11510/) (2023)
- [Personalized Speech Enhancement: New Models and Comprehensive Evaluation](https://mlsanity.com/paper/2110.09625/) (2021)
- [USEV: Universal Speaker Extraction with Visual Cue](https://mlsanity.com/paper/2109.14831/) (2021)
- [A Hybrid Continuity Loss to Reduce Over-Suppression for Time-domain Target Speaker Extraction](https://mlsanity.com/paper/2203.16843/) (2022)
- [Cocktail-Party Audio-Visual Speech Recognition](https://mlsanity.com/paper/2506.02178/) (2025)
- [DeepVQE: Real Time Deep Voice Quality Enhancement for Joint Acoustic Echo Cancellation, Noise Suppression and Dereverberation](https://mlsanity.com/paper/2306.03177/) (2023)
- [Real-Time Audio-Visual End-to-End Speech Enhancement](https://mlsanity.com/paper/2303.07005/) (2023)
- [Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations](https://mlsanity.com/paper/2507.21448/) (2025)
- [Personalized Speech Enhancement Without a Separate Speaker Embedding Model](https://mlsanity.com/paper/2406.09928/) (2024)
- [Online Audio-Visual Autoregressive Speaker Extraction](https://mlsanity.com/paper/2506.01270/) (2025)
- [Plug-and-Steer: Decoupling Separation and Selection in Audio-Visual Target Speaker Extraction](https://mlsanity.com/paper/2603.19697/) (2026)

## Abstract (arXiv)

Online audio-visual target-speaker extraction aims to remove competing voices while preserving speech quality and bounding lookahead. Existing extractors are built and evaluated for separation on synthetic mixtures, leaving listening quality and meeting behavior largely untested. We introduce Audio-Visual Personalized Voice Quality Enhancement (AV-PVQE), which approaches these requirements from the other direction. We start from a personalized speech enhancement model that reconstructs a requested voice at high quality but confuses the target in 46% of two-speaker mixtures despite clean enrollment. Adding mouth features at its speaker-conditioning input and jointly fine-tuning the visual and reconstruction networks reduces this rate to 1.6%, with no future frames and 20 ms of algorithmic delay. Compared with an online autoregressive audio-visual extractor, AV-PVQE yields separation gains on two synthetic benchmarks and larger gains on recorded meetings, and keeps its advantage on excerpts with more speakers than the fine-tuning mixtures. In personalized P.835 listening tests on two meeting corpora, it improves overall quality over this extractor by 0.57 and 0.63 MOS, with similar mean rating relative to the starting model. Preservation and rejection tests show that it keeps the target intact when no competing voice is present and suppresses competing speech when the target is absent.