# Deep speech inpainting of time-frequency masks

arXiv: https://arxiv.org/abs/1910.09058 · HTML: https://mlsanity.com/paper/1910.09058/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2019.

## Built on by (most cited first, 8 of 8)

- [SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detection](https://mlsanity.com/paper/2508.09913/) (2025)
- [Speech inpainting: Context-based speech synthesis guided by video](https://mlsanity.com/paper/2306.00489/) (2023)
- [Enhanced ASR Robustness to Packet Loss with a Front-End Adaptation Network](https://mlsanity.com/paper/2406.18928/) (2024)
- [Speech Recovery for Real-World Self-powered Intermittent Devices](https://mlsanity.com/paper/2106.05229/) (2021)
- [Is Self-Supervised Learning Enough to Fill in the Gap? A Study on Speech Inpainting](https://mlsanity.com/paper/2405.20101/) (2024)
- [Correcting Mispronunciations in Speech using Spectrogram Inpainting](https://mlsanity.com/paper/2204.03379/) (2022)
- [Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach](https://mlsanity.com/paper/2406.00901/) (2024)
- [Sequence-to-Sequence Multi-Modal Speech In-Painting](https://mlsanity.com/paper/2406.01321/) (2024)

## Abstract (arXiv)

Transient loud intrusions, often occurring in noisy environments, can completely overpower speech signal and lead to an inevitable loss of information. While existing algorithms for noise suppression can yield impressive results, their efficacy remains limited for very low signal-to-noise ratios or when parts of the signal are missing. To address these limitations, here we propose an end-to-end framework for speech inpainting, the context-based retrieval of missing or severely distorted parts of time-frequency representation of speech. The framework is based on a convolutional U-Net trained via deep feature losses, obtained using speechVGG, a deep speech feature extractor pre-trained on an auxiliary word classification task. Our evaluation results demonstrate that the proposed framework can recover large portions of missing or distorted time-frequency representation of speech, up to 400 ms and 3.2 kHz in bandwidth. In particular, our approach provided a substantial increase in STOI & PESQ objective metrics of the initially corrupted speech samples. Notably, using deep feature losses to train the framework led to the best results, as compared to conventional approaches.