# My lips are concealed: Audio-visual speech enhancement through obstructions

arXiv: https://arxiv.org/abs/1907.04975 · HTML: https://mlsanity.com/paper/1907.04975/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2019.

## Built on by (most cited first, 26 of 26)

- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Move2Hear: Active Audio-Visual Source Separation](https://mlsanity.com/paper/2105.07142/) (2021)
- [The Right to Talk: An Audio-Visual Transformer Approach](https://mlsanity.com/paper/2108.03256/) (2021)
- [iQuery: Instruments as Queries for Audio-Visual Sound Separation](https://mlsanity.com/paper/2212.03814/) (2022)
- [Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation](https://mlsanity.com/paper/2104.02775/) (2021)
- [Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis](https://mlsanity.com/paper/2203.17263/) (2022)
- [Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video](https://mlsanity.com/paper/2111.10882/) (2021)
- [Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser](https://mlsanity.com/paper/2305.17343/) (2023)
- [Active Audio-Visual Separation of Dynamic Sound Sources](https://mlsanity.com/paper/2202.00850/) (2022)
- [AdVerb: Visually Guided Audio Dereverberation](https://mlsanity.com/paper/2308.12370/) (2023)
- [Look Once to Hear: Target Speech Hearing with Noisy Examples](https://mlsanity.com/paper/2405.06289/) (2024)
- [Multimodal Attention Fusion for Target Speaker Extraction](https://mlsanity.com/paper/2102.01326/) (2021)
- [Diffusion-based Unsupervised Audio-visual Speech Enhancement](https://mlsanity.com/paper/2410.05301/) (2024)
- [Switching Variational Auto-Encoders for Noise-Agnostic Audio-visual Speech Enhancement](https://mlsanity.com/paper/2102.04144/) (2021)
- [WASE: Learning When to Attend for Speaker Extraction in Cocktail Party Environments](https://mlsanity.com/paper/2106.07016/) (2021)
- [The impact of removing head movements on audio-visual speech enhancement](https://mlsanity.com/paper/2202.00538/) (2022)
- [Large-scale unsupervised audio pre-training for video-to-speech synthesis](https://mlsanity.com/paper/2306.15464/) (2023)
- [Towards Robust Real-time Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2112.09060/) (2021)
- [Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction](https://mlsanity.com/paper/2207.04213/) (2022)
- [AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modeling](https://mlsanity.com/paper/2406.11619/) (2024)
- [MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues](https://mlsanity.com/paper/2412.08247/) (2024)
- [Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2508.13624/) (2025)
- [Lightweight Wasserstein Audio-Visual Model for Unified Speech Enhancement and Separation](https://mlsanity.com/paper/2512.06689/) (2025)
- [Visual-Informed Speech Enhancement Using Attention-Based Beamforming](https://mlsanity.com/paper/2603.05270/) (2026)
- [Automatic Contextual Audio Denoising](https://mlsanity.com/paper/2605.22262/) (2026)

## Abstract (arXiv)

Our objective is an audio-visual model for separating a single speaker from a mixture of sounds such as other speakers and background noise. Moreover, we wish to hear the speaker even when the visual cues are temporarily absent due to occlusion. To this end we introduce a deep audio-visual speech enhancement network that is able to separate a speaker's voice by conditioning on both the speaker's lip movements and/or a representation of their voice. The voice representation can be obtained by either (i) enrollment, or (ii) by self-enrollment -- learning the representation on-the-fly given sufficient unobstructed visual input. The model is trained by blending audios, and by introducing artificial occlusions around the mouth region that prevent the visual modality from dominating. The method is speaker-independent, and we demonstrate it on real examples of speakers unheard (and unseen) during training. The method also improves over previous models in particular for cases of occlusion in the visual modality.