mlsanityData to 7 Oct 2026

High Fidelity Speech Regeneration with Application to Speech Enhancement

arXiv 2102.00429 ↗First posted 31 Jan 2021cs.SD cs.LG eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

Speech enhancement has seen great improvement in recent years mainly through contributions in denoising, speaker separation, and dereverberation methods that mostly deal with environmental effects on vocal audio. To enhance speech beyond the limitations of the original signal, we take a regeneration approach, in which we recreate the speech from its essence, including the semi-recognized speech, prosody features, and identity. We propose a wav-to-wav generative model for speech that can generate 24khz speech in a real-time manner and which utilizes a compact speech representation, composed of ASR and identity features, to achieve a higher level of intelligibility. Inspired by voice conversion methods, we train to augment the speech characteristics while preserving the identity of the source using an auxiliary identity network. Perceptual acoustic metrics and subjective tests show that the method obtains valuable improvements over recent baselines.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 19 of its 35 references that are in the map. First posted on arXiv 31 Jan 2021.

Builds on

19 of 35 references.

References · 19

  1. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  2. 2016Least Squares Generative Adversarial Networkscited by 1,000
  3. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  4. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  5. 2016A Learned Representation For Artistic Stylecited by 340
  6. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  7. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  8. 2020Real Time Speech Enhancement in the Waveform Domaincited by 183
  9. 2018Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separationcited by 125
  10. 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 104
  11. 2019High Fidelity Speech Synthesis with Adversarial Networkscited by 90
  12. 2019Jasper: An End-to-End Convolutional Neural Acoustic Modelcited by 75
7 more
  1. 2017A Wavenet for Speech Denoisingcited by 47
  2. 2019A scalable noisy speech dataset and online subjective test frameworkcited by 47
  3. 2016Wav2Letter: an End-to-End ConvNet-based Speech Recognition Systemcited by 43
  4. 2020A Spectral Energy Distance for Parallel Speech Synthesiscited by 42
  5. 2020HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networkscited by 40
  6. 2019TTS Skins: Speaker Conversion via ASRcited by 14
  7. 2019High-quality Speech Synthesis Using Super-resolution Mel-Spectrogramcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 4; 2023: 1; 2024: 1.

10 papers.

  1. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  2. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  3. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  4. 2023The Ethical Implications of Generative Audio Models: A Systematic Literature Reviewcited by 18
  5. 2022Music Enhancement via Image Translation and Vocodingcited by 11
  6. 2022Self-Supervised Learning for Speech Enhancement through Synthesiscited by 10
  7. 2021Direct Simultaneous Speech-to-Speech Translation with Variational Monotonic Multihead Attentioncited by 9
  8. 2022ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancementcited by 8
  9. 2024V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flowcited by 5
  10. 2021A Training Framework for Stereo-Aware Speech Enhancement using Deep Neural Networkscited by 1