mlsanityData to 7 Oct 2026

Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility

arXiv 2409.09357 ↗First posted 14 Sep 2024cs.SD cs.AI eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Speech restoration aims at restoring full-band speech with high quality and intelligibility, considering a diverse set of distortions. MaskSR is a recently proposed generative model for this task. As other models of its kind, MaskSR attains high quality but, as we show, intelligibility can be substantially improved. We do so by boosting the speech encoder component of MaskSR with predictions of semantic representations of the target speech, using a pre-trained self-supervised teacher model. Then, a masked language model is conditioned on the learned semantic features to predict acoustic tokens that encode low level spectral details of the target speech. We show that, with the same MaskSR model capacity and inference time, the proposed model, MaskSR2, significantly reduces the word error rate, a typical metric for intelligibility. MaskSR2 also achieves competitive word error rate among other models, while providing superior quality. An ablation study shows the effectiveness of various semantic representations.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 39 of its 43 references that are in the map. First posted on arXiv 14 Sep 2024.

Builds on

39 of 43 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  7. 2023MusicLM: Generating Music From Textcited by 442
  8. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  9. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  10. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  11. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  12. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
27 more
  1. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Modelscited by 195
  2. 2020Real Time Speech Enhancement in the Waveform Domaincited by 183
  3. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  4. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Resultscited by 128
  5. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberationcited by 105
  6. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 92
  7. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  8. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  9. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  10. 2022Improved Masked Image Generation with Token-Criticcited by 63
  11. 2019WHAMR!: Noisy and Reverberant Single-Channel Speech Separationcited by 61
  12. 2022ICASSP 2022 Deep Noise Suppression Challengecited by 57
  13. 2021VoiceFixer: Toward General Speech Restoration with Neural Vocodercited by 50
  14. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  15. 2023SELM: Speech Enhancement Using Discrete Tokens and Language Modelscited by 36
  16. 2023DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancementcited by 25
  17. 2022FRCRN: Boosting Feature Representation using Frequency Recurrence for Monaural Speech Enhancementcited by 24
  18. 2024MaskSR: Masked Language Model for Full-band Speech Restorationcited by 24
  19. 2024Universal Score-based Speech Enhancement with High Content Preservationcited by 23
  20. 2023Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representationscited by 22
  21. 2023CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Modelscited by 20
  22. 2020SESQA: semi-supervised learning for speech quality assessmentcited by 13
  23. 2021WARP-Q: Quality Prediction For Generative Neural Speech Codecscited by 12
  24. 2021A Simultaneous Denoising and Dereverberation Framework with Target Decouplingcited by 11
  25. 2022Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillationcited by 8
  26. 2023Low-Resource Self-Supervised Learning with SSL-Enhanced TTScited by 4
  27. 2023Gesper: A Restoration-Enhancement Framework for General Speech Reconstructioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  2. 2025AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancementcited by 23
  3. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  4. 2025Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restorationcited by 1
  5. 2026Absorbing Discrete Diffusion for Speech Enhancementcited by 0