# Audio-Visual Speech Inpainting with Deep Learning

arXiv: https://arxiv.org/abs/2010.04556 · HTML: https://mlsanity.com/paper/2010.04556/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2020.

## Built on by (most cited first, 10 of 10)

- [Can audio-visual integration strengthen robustness under multimodal attacks?](https://mlsanity.com/paper/2104.02000/) (2021)
- [SpeechPainter: Text-conditioned Speech Inpainting](https://mlsanity.com/paper/2202.07273/) (2022)
- [ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement](https://mlsanity.com/paper/2212.11377/) (2022)
- [Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos](https://mlsanity.com/paper/2307.04760/) (2023)
- [Speech inpainting: Context-based speech synthesis guided by video](https://mlsanity.com/paper/2306.00489/) (2023)
- [Speech Recovery for Real-World Self-powered Intermittent Devices](https://mlsanity.com/paper/2106.05229/) (2021)
- [Is Self-Supervised Learning Enough to Fill in the Gap? A Study on Speech Inpainting](https://mlsanity.com/paper/2405.20101/) (2024)
- [Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach](https://mlsanity.com/paper/2406.00901/) (2024)
- [Sequence-to-Sequence Multi-Modal Speech In-Painting](https://mlsanity.com/paper/2406.01321/) (2024)
- [WaveLLDM: Design and Development of a Lightweight Latent Diffusion Model for Speech Enhancement and Restoration](https://mlsanity.com/paper/2508.21153/) (2025)

## Abstract (arXiv)

In this paper, we present a deep-learning-based framework for audio-visual speech inpainting, i.e., the task of restoring the missing parts of an acoustic speech signal from reliable audio context and uncorrupted visual information. Recent work focuses solely on audio-only methods and generally aims at inpainting music signals, which show highly different structure than speech. Instead, we inpaint speech signals with gaps ranging from 100 ms to 1600 ms to investigate the contribution that vision can provide for gaps of different duration. We also experiment with a multi-task learning approach where a phone recognition task is learned together with speech inpainting. Results show that the performance of audio-only speech inpainting approaches degrades rapidly when gaps get large, while the proposed audio-visual approach is able to plausibly restore missing information. In addition, we show that multi-task learning is effective, although the largest contribution to performance comes from vision.