# AV-SepFormer: Cross-Attention SepFormer for Audio-Visual Target Speaker Extraction

arXiv: https://arxiv.org/abs/2306.14170 · HTML: https://mlsanity.com/paper/2306.14170/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2023.

## Built on by (most cited first, 15 of 15)

- [Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2404.12725/) (2024)
- [Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction](https://mlsanity.com/paper/2310.07284/) (2023)
- [Target conversation extraction: Source separation using turn-taking dynamics](https://mlsanity.com/paper/2407.11277/) (2024)
- [Cross-attention Inspired Selective State Space Models for Target Sound Extraction](https://mlsanity.com/paper/2409.04803/) (2024)
- [RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation](https://mlsanity.com/paper/2309.17189/) (2023)
- [C2AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction](https://mlsanity.com/paper/2504.00750/) (2025)
- [Plug-and-Play Co-Occurring Face Attention for Robust Audio-Visual Speaker Extraction](https://mlsanity.com/paper/2505.20635/) (2025)
- [Neural Speech Extraction with Human Feedback](https://mlsanity.com/paper/2508.03041/) (2025)
- [TDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down Fusion](https://mlsanity.com/paper/2401.14185/) (2024)
- [AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modeling](https://mlsanity.com/paper/2406.11619/) (2024)
- [SNIFR : Boosting Fine-Grained Child Harmful Content Detection Through Audio-Visual Alignment with Cascaded Cross-Transformer](https://mlsanity.com/paper/2506.03378/) (2025)
- [Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2506.09792/) (2025)
- [GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection](https://mlsanity.com/paper/2512.15707/) (2025)
- [Noise-Robust Target-Speaker Voice Activity Detection Through Self-Supervised Pretraining](https://mlsanity.com/paper/2501.03184/) (2025)
- [ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction](https://mlsanity.com/paper/2511.06288/) (2025)

## Abstract (arXiv)

Visual information can serve as an effective cue for target speaker extraction (TSE) and is vital to improving extraction performance. In this paper, we propose AV-SepFormer, a SepFormer-based attention dual-scale model that utilizes cross- and self-attention to fuse and model features from audio and visual. AV-SepFormer splits the audio feature into a number of chunks, equivalent to the length of the visual feature. Then self- and cross-attention are employed to model and fuse the multi-modal features. Furthermore, we use a novel 2D positional encoding, that introduces the positional information between and within chunks and provides significant gains over the traditional positional encoding. Our model has two key advantages: the time granularity of audio chunked feature is synchronized to the visual feature, which alleviates the harm caused by the inconsistency of audio and video sampling rate; by combining self- and cross-attention, feature fusion and speech extraction processes are unified within an attention paradigm. The experimental results show that AV-SepFormer significantly outperforms other existing methods.