# Time Domain Audio Visual Speech Separation

arXiv: https://arxiv.org/abs/1904.03760 · HTML: https://mlsanity.com/paper/1904.03760/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2019.

## Built on by (most cited first, 30 of 30)

- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer](https://mlsanity.com/paper/2203.04099/) (2022)
- [VisualTTS: TTS with Accurate Lip-Speech Synchronization for Automatic Voice Over](https://mlsanity.com/paper/2110.03342/) (2021)
- [A cappella: Audio-visual Singing Voice Separation](https://mlsanity.com/paper/2104.09946/) (2021)
- [IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation](https://mlsanity.com/paper/2308.08143/) (2023)
- [Multimodal Attention Fusion for Target Speaker Extraction](https://mlsanity.com/paper/2102.01326/) (2021)
- [An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits](https://mlsanity.com/paper/2212.10744/) (2022)
- [Multi-Channel Multi-Speaker ASR Using 3D Spatial Feature](https://mlsanity.com/paper/2111.11023/) (2021)
- [Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2404.12725/) (2024)
- [X-SepFormer: End-to-end Speaker Extraction Network with Explicit Optimization on Speaker Confusion](https://mlsanity.com/paper/2303.05023/) (2023)
- [Speaker Extraction with Co-Speech Gestures Cue](https://mlsanity.com/paper/2203.16840/) (2022)
- [Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model](https://mlsanity.com/paper/2306.00160/) (2023)
- [Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition](https://mlsanity.com/paper/2307.02909/) (2023)
- [Audio-visual multi-channel speech separation, dereverberation and recognition](https://mlsanity.com/paper/2204.01977/) (2022)
- [RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation](https://mlsanity.com/paper/2309.17189/) (2023)
- [Rethinking Audio-Visual Adversarial Vulnerability from Temporal and Modality Perspectives](https://mlsanity.com/paper/2502.11858/) (2025)
- [C2AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction](https://mlsanity.com/paper/2504.00750/) (2025)
- [Plug-and-Play Co-Occurring Face Attention for Robust Audio-Visual Speaker Extraction](https://mlsanity.com/paper/2505.20635/) (2025)
- [Efficient Audio-Visual Speech Separation with Discrete Lip Semantics and Multi-Scale Global-Local Attention](https://mlsanity.com/paper/2509.23610/) (2025)
- [Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction](https://mlsanity.com/paper/2207.04213/) (2022)
- [Audio-visual video-to-speech synthesis with synthesized input audio](https://mlsanity.com/paper/2307.16584/) (2023)
- [TDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down Fusion](https://mlsanity.com/paper/2401.14185/) (2024)
- [AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modeling](https://mlsanity.com/paper/2406.11619/) (2024)
- [CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization](https://mlsanity.com/paper/2505.03186/) (2025)
- [Multi-Modal Multi-Correlation Learning for Audio-Visual Speech Separation](https://mlsanity.com/paper/2207.01197/) (2022)
- [VCSE: Time-Domain Visual-Contextual Speaker Extraction Network](https://mlsanity.com/paper/2210.06177/) (2022)
- [RIR-SF: Room Impulse Response Based Spatial Feature for Target Speech Recognition in Multi-Channel Multi-Speaker Scenarios](https://mlsanity.com/paper/2311.00146/) (2023)
- [Audio-Visual Speech Enhancement In Complex Scenarios With Separation And Dereverberation Joint Modeling](https://mlsanity.com/paper/2510.26825/) (2025)
- [Lightweight Wasserstein Audio-Visual Model for Unified Speech Enhancement and Separation](https://mlsanity.com/paper/2512.06689/) (2025)
- [DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation](https://mlsanity.com/paper/2608.09288/) (2026)

## Abstract (arXiv)

Audio-visual multi-modal modeling has been demonstrated to be effective in many speech related tasks, such as speech recognition and speech enhancement. This paper introduces a new time-domain audio-visual architecture for target speaker extraction from monaural mixtures. The architecture generalizes the previous TasNet (time-domain speech separation network) to enable multi-modal learning and at meanwhile it extends the classical audio-visual speech separation from frequency-domain to time-domain. The main components of proposed architecture include an audio encoder, a video encoder that extracts lip embedding from video streams, a multi-modal separation network and an audio decoder. Experiments on simulated mixtures based on recently released LRS2 dataset show that our method can bring 3dB+ and 4dB+ Si-SNR improvements on two- and three-speaker cases respectively, compared to audio-only TasNet and frequency-domain audio-visual networks