# Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement

arXiv: https://arxiv.org/abs/2508.13624 · HTML: https://mlsanity.com/paper/2508.13624/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Speech and audio SSMs, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 8 of its 9 references that are in the map.
- First posted on arXiv 19 Aug 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Speech and audio SSMs](https://mlsanity.com/ssm/speech-and-audio-ssms/)

## Builds on (most cited first, 8 of 8)

- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [An Investigation of Incorporating Mamba for Speech Enhancement](https://mlsanity.com/paper/2405.06573/) (2024)
- [My lips are concealed: Audio-visual speech enhancement through obstructions](https://mlsanity.com/paper/1907.04975/) (2019)
- [FaceFilter: Audio-visual speech separation using still images](https://mlsanity.com/paper/2005.07074/) (2020)
- [Audio-Visual Speech Enhancement and Separation by Utilizing Multi-Modal Self-Supervised Embeddings](https://mlsanity.com/paper/2210.17456/) (2022)
- [Universal Speech Enhancement with Regression and Generative Mamba](https://mlsanity.com/paper/2505.21198/) (2025)
- [SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model](https://mlsanity.com/paper/2411.07751/) (2024)
- [Deep Complex U-Net with Conformer for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2309.11059/) (2023)

## Abstract (arXiv)

Recent Mamba-based models have shown promise in speech enhancement by efficiently modeling long-range temporal dependencies. However, models like Speech Enhancement Mamba (SEMamba) remain limited to single-speaker scenarios and struggle in complex multi-speaker environments such as the cocktail party problem. To overcome this, we introduce AVSEMamba, an audio-visual speech enhancement model that integrates full-face visual cues with a Mamba-based temporal backbone. By leveraging spatiotemporal visual information, AVSEMamba enables more accurate extraction of target speech in challenging conditions. Evaluated on the AVSEC-4 Challenge development and blind test sets, AVSEMamba outperforms other monaural baselines in speech intelligibility (STOI), perceptual quality (PESQ), and non-intrusive quality (UTMOS), and achieves 1st place on the monaural leaderboard.