# Speech inpainting: Context-based speech synthesis guided by video

arXiv: https://arxiv.org/abs/2306.00489 · HTML: https://mlsanity.com/paper/2306.00489/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 23 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020)
- [Audio-Visual Synchronisation in the wild](https://mlsanity.com/paper/2112.04432/) (2021)
- [VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices](https://mlsanity.com/paper/2204.02090/) (2022)
- [SpeechPainter: Text-conditioned Speech Inpainting](https://mlsanity.com/paper/2202.07273/) (2022)
- [VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer](https://mlsanity.com/paper/2203.04099/) (2022)
- [Audio-Visual Speech Inpainting with Deep Learning](https://mlsanity.com/paper/2010.04556/) (2020)
- [Deep speech inpainting of time-frequency masks](https://mlsanity.com/paper/1910.09058/) (2019)
- [Deep Long Audio Inpainting](https://mlsanity.com/paper/1911.06476/) (2019)
- [Audio inpainting with generative adversarial network](https://mlsanity.com/paper/2003.07704/) (2020)

## Built on by (most cited first, 4 of 4)

- [Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach](https://mlsanity.com/paper/2406.00901/) (2024)
- [Learning from Silence and Noise for Visual Sound Source Localization](https://mlsanity.com/paper/2508.21761/) (2025)
- [Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios](https://mlsanity.com/paper/2603.08249/) (2026)
- [Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing](https://mlsanity.com/paper/2608.06424/) (2026)

## Abstract (arXiv)

Audio and visual modalities are inherently connected in speech signals: lip movements and facial expressions are correlated with speech sounds. This motivates studies that incorporate the visual modality to enhance an acoustic speech signal or even restore missing audio information. Specifically, this paper focuses on the problem of audio-visual speech inpainting, which is the task of synthesizing the speech in a corrupted audio segment in a way that it is consistent with the corresponding visual content and the uncorrupted audio context. We present an audio-visual transformer-based deep learning model that leverages visual cues that provide information about the content of the corrupted audio. It outperforms the previous state-of-the-art audio-visual model and audio-only baselines. We also show how visual features extracted with AV-HuBERT, a large audio-visual transformer for speech recognition, are suitable for synthesizing speech.