mlsanityData to 7 Oct 2026

MaskSR: Masked Language Model for Full-band Speech Restoration

arXiv 2406.02092 ↗First posted 4 Jun 2024cs.SD cs.AI cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Speech restoration aims at restoring high quality speech in the presence of a diverse set of distortions. Although several deep learning paradigms have been studied for this task, the power of the recently emerging language models has not been fully explored. In this paper, we propose MaskSR, a masked language model capable of restoring full-band 44.1 kHz speech jointly considering noise, reverb, clipping, and low bandwidth. MaskSR works with discrete acoustic tokens extracted using a pre-trained neural codec. During training, MaskSR is optimized to predict randomly masked tokens extracted from the high quality target speech, conditioned on the corrupted speech with various distortions. During inference, MaskSR reconstructs the target speech tokens with efficient iterative sampling. Extensive experiments show that MaskSR obtains competitive results on both the full-band speech restoration task and also on sub-tasks compared with a wide range of models.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 37 of its 40 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

37 of 40 references.

References · 37

  1. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  2. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  3. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  4. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  5. 2023Simple and Controllable Music Generationcited by 480
  6. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  7. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  8. 2022AudioGen: Textually Guided Audio Generationcited by 292
  9. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
  10. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  11. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Modelscited by 195
  12. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Resultscited by 128
25 more
  1. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  2. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  3. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberationcited by 105
  4. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 92
  5. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  6. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  7. 2020FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancementcited by 66
  8. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  9. 2022ICASSP 2022 Deep Noise Suppression Challengecited by 57
  10. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  11. 2021VoiceFixer: Toward General Speech Restoration with Neural Vocodercited by 50
  12. 2022VoiceFixer: A Unified Framework for High-Fidelity Speech Restorationcited by 37
  13. 2023SELM: Speech Enhancement Using Discrete Tokens and Language Modelscited by 36
  14. 2022FRCRN: Boosting Feature Representation using Frequency Recurrence for Monaural Speech Enhancementcited by 24
  15. 2020Data augmentation and loss normalization for deep noise suppressioncited by 23
  16. 2023Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representationscited by 22
  17. 2020SESQA: semi-supervised learning for speech quality assessmentcited by 13
  18. 2021WARP-Q: Quality Prediction For Generative Neural Speech Codecscited by 12
  19. 2021A Simultaneous Denoising and Dereverberation Framework with Target Decouplingcited by 11
  20. 2022Music Enhancement via Image Translation and Vocodingcited by 11
  21. 2021Adversarial Auto-Encoding for Packet Loss Concealmentcited by 5
  22. 2023Gesper: A Restoration-Enhancement Framework for General Speech Reconstructioncited by 3
  23. 2023Mono-to-stereo through parametric stereo generationcited by 3
  24. 2024RaD-Net: A Repairing and Denoising Network for Speech Signal Improvementcited by 3
  25. 2023Two-stage Neural Network for ICASSP 2023 Speech Signal Improvement Challengecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 6.

24 papers.

Sort:

By area:

  1. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  2. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 109
  3. 2025AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancementcited by 23
  4. 2025LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancementcited by 19
  5. 2025FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matchingcited by 17
  6. 2024TSELM: Target Speaker Extraction using Discrete Tokens and Language Modelscited by 10
  7. 2025Speech Enhancement Using Continuous Embeddings of Neural Audio Codeccited by 7
  8. 2025Universal Speech Enhancement with Regression and Generative Mambacited by 7
  9. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  10. 2025DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformerscited by 6
  11. 2025SonicMaster: Towards Controllable All-in-One Music Restoration and Masteringcited by 6
  12. 2024Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibilitycited by 5
12 more
  1. 2026FlowSE-GRPO: Training Flow Matching Speech Enhancement via Online Reinforcement Learningcited by 5
  2. 2025VoiceBridge: General Speech Restoration with One-step Latent Bridge Modelscited by 4
  3. 2025UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancementcited by 4
  4. 2025SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancementcited by 3
  5. 2025MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancementcited by 2
  6. 2025Aligning Generative Speech Enhancement with Perceptual Feedbackcited by 2
  7. 2025High-Fidelity Speech Enhancement via Discrete Audio Tokenscited by 2
  8. 2026Post-Training Speech Enhancement Language Models with Perceptual Rewardscited by 1
  9. 2026Absorbing Discrete Diffusion for Speech Enhancementcited by 0
  10. 2026MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Ratescited by 0
  11. 2026UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinationscited by 0
  12. 2026Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representationscited by 0