# Sequence-to-Sequence Multi-Modal Speech In-Painting

arXiv: https://arxiv.org/abs/2406.01321 · HTML: https://mlsanity.com/paper/2406.01321/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [LipNet: End-to-End Sentence-level Lipreading](https://mlsanity.com/paper/1611.01599/) (2016)
- [Vision-Infused Deep Audio Inpainting](https://mlsanity.com/paper/1910.10997/) (2019)
- [Learn to Combine Modalities in Multimodal Deep Learning](https://mlsanity.com/paper/1805.11730/) (2018)
- [Lipreading with Long Short-Term Memory](https://mlsanity.com/paper/1601.08188/) (2016)
- [GACELA -- A generative adversarial context encoder for long audio inpainting](https://mlsanity.com/paper/2005.05032/) (2020)
- [Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environments](https://mlsanity.com/paper/1811.02480/) (2018)
- [Inpainting of long audio segments with similarity graphs](https://mlsanity.com/paper/1607.06667/) (2016)
- [Audio-Visual Speech Inpainting with Deep Learning](https://mlsanity.com/paper/2010.04556/) (2020)
- [Deep speech inpainting of time-frequency masks](https://mlsanity.com/paper/1910.09058/) (2019)
- [Deep Long Audio Inpainting](https://mlsanity.com/paper/1911.06476/) (2019)
- [Audio inpainting with generative adversarial network](https://mlsanity.com/paper/2003.07704/) (2020)
- [Audio Inpainting: Revisited and Reweighted](https://mlsanity.com/paper/2001.02480/) (2020)

## Abstract (arXiv)

Speech in-painting is the task of regenerating missing audio contents using reliable context information. Despite various recent studies in multi-modal perception of audio in-painting, there is still a need for an effective infusion of visual and auditory information in speech in-painting. In this paper, we introduce a novel sequence-to-sequence model that leverages the visual information to in-paint audio signals via an encoder-decoder architecture. The encoder plays the role of a lip-reader for facial recordings and the decoder takes both encoder outputs as well as the distorted audio spectrograms to restore the original speech. Our model outperforms an audio-only speech in-painting model and has comparable results with a recent multi-modal speech in-painter in terms of speech quality and intelligibility metrics for distortions of 300 ms to 1500 ms duration, which proves the effectiveness of the introduced multi-modality in speech in-painting.