# FaceFilter: Audio-visual speech separation using still images

arXiv: https://arxiv.org/abs/2005.07074 · HTML: https://mlsanity.com/paper/2005.07074/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2020.

## Built on by (most cited first, 25 of 25)

- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Move2Hear: Active Audio-Visual Source Separation](https://mlsanity.com/paper/2105.07142/) (2021)
- [Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation](https://mlsanity.com/paper/2104.02775/) (2021)
- [Imaginary Voice: Face-styled Diffusion Model for Text-to-Speech](https://mlsanity.com/paper/2302.13700/) (2023)
- [Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis](https://mlsanity.com/paper/2203.17263/) (2022)
- [Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video](https://mlsanity.com/paper/2111.10882/) (2021)
- [Look Who's Talking: Active Speaker Detection in the Wild](https://mlsanity.com/paper/2108.07640/) (2021)
- [Active Audio-Visual Separation of Dynamic Sound Sources](https://mlsanity.com/paper/2202.00850/) (2022)
- [VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer](https://mlsanity.com/paper/2203.04099/) (2022)
- [A cappella: Audio-visual Singing Voice Separation](https://mlsanity.com/paper/2104.09946/) (2021)
- [LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading](https://mlsanity.com/paper/2112.04748/) (2021)
- [Continual Audio-Visual Sound Separation](https://mlsanity.com/paper/2411.02860/) (2024)
- [Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast](https://mlsanity.com/paper/2204.14057/) (2022)
- [Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models](https://mlsanity.com/paper/2306.16003/) (2023)
- [Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition](https://mlsanity.com/paper/2307.02909/) (2023)
- [WASE: Learning When to Attend for Speaker Extraction in Cocktail Party Environments](https://mlsanity.com/paper/2106.07016/) (2021)
- [Disentangling the Impacts of Language and Channel Variability on Speech Separation Networks](https://mlsanity.com/paper/2203.16040/) (2022)
- [SonoWorld: From One Image to a 3D Audio-Visual Scene](https://mlsanity.com/paper/2603.28757/) (2026)
- [Robust Active Speaker Detection in Noisy Environments](https://mlsanity.com/paper/2403.19002/) (2024)
- [Egocentric Audio-Visual Noise Suppression](https://mlsanity.com/paper/2211.03643/) (2022)
- [AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modeling](https://mlsanity.com/paper/2406.11619/) (2024)
- [Multi-Modal Multi-Correlation Learning for Audio-Visual Speech Separation](https://mlsanity.com/paper/2207.01197/) (2022)
- [MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues](https://mlsanity.com/paper/2412.08247/) (2024)
- [Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2508.13624/) (2025)

## Abstract (arXiv)

The objective of this paper is to separate a target speaker's speech from a mixture of two speakers using a deep audio-visual speech separation network. Unlike previous works that used lip movement on video clips or pre-enrolled speaker information as an auxiliary conditional feature, we use a single face image of the target speaker. In this task, the conditional feature is obtained from facial appearance in cross-modal biometric task, where audio and visual identity representations are shared in latent space. Learnt identities from facial images enforce the network to isolate matched speakers and extract the voices from mixed speech. It solves the permutation problem caused by swapped channel outputs, frequently occurred in speech separation tasks. The proposed method is far more practical than video-based speech separation since user profile images are readily available on many platforms. Also, unlike speaker-aware separation methods, it is applicable on separation with unseen speakers who have never been enrolled before. We show strong qualitative and quantitative results on challenging real-world examples.