# Robust Unsupervised Audio-visual Speech Enhancement Using a Mixture of Variational Autoencoders

arXiv: https://arxiv.org/abs/1911.03930 · HTML: https://mlsanity.com/paper/1911.03930/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 8 of 8)

- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [Learning in Audio-visual Context: A Review, Analysis, and New Perspective](https://mlsanity.com/paper/2208.09579/) (2022)
- [Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser](https://mlsanity.com/paper/2305.17343/) (2023)
- [Unsupervised speech enhancement with diffusion-based generative models](https://mlsanity.com/paper/2309.10450/) (2023)
- [Switching Variational Auto-Encoders for Noise-Agnostic Audio-visual Speech Enhancement](https://mlsanity.com/paper/2102.04144/) (2021)
- [The impact of removing head movements on audio-visual speech enhancement](https://mlsanity.com/paper/2202.00538/) (2022)
- [Audio-visual speech enhancement with a deep Kalman filter generative model](https://mlsanity.com/paper/2211.00988/) (2022)
- [MoMuSE: Momentum Multi-modal Target Speaker Extraction for Real-time Scenarios with Impaired Visual Cues](https://mlsanity.com/paper/2412.08247/) (2024)

## Abstract (arXiv)

Recently, an audio-visual speech generative model based on variational autoencoder (VAE) has been proposed, which is combined with a nonnegative matrix factorization (NMF) model for noise variance to perform unsupervised speech enhancement. When visual data is clean, speech enhancement with audio-visual VAE shows a better performance than with audio-only VAE, which is trained on audio-only data. However, audio-visual VAE is not robust against noisy visual data, e.g., when for some video frames, speaker face is not frontal or lips region is occluded. In this paper, we propose a robust unsupervised audio-visual speech enhancement method based on a per-frame VAE mixture model. This mixture model consists of a trained audio-only VAE and a trained audio-visual VAE. The motivation is to skip noisy visual frames by switching to the audio-only VAE model. We present a variational expectation-maximization method to estimate the parameters of the model. Experiments show the promising performance of the proposed method.