# Enhancing Gappy Speech Audio Signals with Generative Adversarial Networks

arXiv: https://arxiv.org/abs/2305.05780 · HTML: https://mlsanity.com/paper/2305.05780/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 11 of 11)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [Image Inpainting via Generative Multi-column Convolutional Neural Networks](https://mlsanity.com/paper/1810.08771/) (2018)
- [Vision-Infused Deep Audio Inpainting](https://mlsanity.com/paper/1910.10997/) (2019)
- [Improving GANs for Speech Enhancement](https://mlsanity.com/paper/2001.05532/) (2020)
- [R-MNet: A Perceptual Adversarial Network for Image Inpainting](https://mlsanity.com/paper/2008.04621/) (2020)
- [Fast video object segmentation with Spatio-Temporal GANs](https://mlsanity.com/paper/1903.12161/) (2019)

## Abstract (arXiv)

Gaps, dropouts and short clips of corrupted audio are a common problem and particularly annoying when they occur in speech. This paper uses machine learning to regenerate gaps of up to 320ms in an audio speech signal. Audio regeneration is translated into image regeneration by transforming audio into a Mel-spectrogram and using image in-painting to regenerate the gaps. The full Mel-spectrogram is then transferred back to audio using the Parallel-WaveGAN vocoder and integrated into the audio stream. Using a sample of 1300 spoken audio clips of between 1 and 10 seconds taken from the publicly-available LJSpeech dataset our results show regeneration of audio gaps in close to real time using GANs with a GPU equipped system. As expected, the smaller the gap in the audio, the better the quality of the filled gaps. On a gap of 240ms the average mean opinion score (MOS) for the best performing models was 3.737, on a scale of 1 (worst) to 5 (best) which is sufficient for a human to perceive as close to uninterrupted human speech.