# MaskSR: Masked Language Model for Full-band Speech Restoration

arXiv: https://arxiv.org/abs/2406.02092 · HTML: https://mlsanity.com/paper/2406.02092/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 40 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 30 of 37)

- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [UniAudio: An Audio Foundation Model Toward Universal Audio Generation](https://mlsanity.com/paper/2310.00704/) (2023)
- [SoundStorm: Efficient Parallel Audio Generation](https://mlsanity.com/paper/2305.09636/) (2023)
- [StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2212.11851/) (2022)
- [Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkit](https://mlsanity.com/paper/2210.17016/) (2022)
- [I Hear Your True Colors: Image Guided Audio Generation](https://mlsanity.com/paper/2211.03089/) (2022)
- [Universal Speech Enhancement with Score-based Diffusion](https://mlsanity.com/paper/2206.03065/) (2022)
- [FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement](https://mlsanity.com/paper/2010.15508/) (2020)
- [SpeechX: Neural Codec Language Model as a Versatile Speech Transformer](https://mlsanity.com/paper/2308.06873/) (2023)
- [ICASSP 2022 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2202.13288/) (2022)
- [VampNet: Music Generation via Masked Acoustic Token Modeling](https://mlsanity.com/paper/2307.04686/) (2023)
- [VoiceFixer: Toward General Speech Restoration with Neural Vocoder](https://mlsanity.com/paper/2109.13731/) (2021)
- [VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration](https://mlsanity.com/paper/2204.05841/) (2022)
- [SELM: Speech Enhancement Using Discrete Tokens and Language Models](https://mlsanity.com/paper/2312.09747/) (2023)
- [FRCRN: Boosting Feature Representation using Frequency Recurrence for Monaural Speech Enhancement](https://mlsanity.com/paper/2206.07293/) (2022)
- [Data augmentation and loss normalization for deep noise suppression](https://mlsanity.com/paper/2008.06412/) (2020)
- [Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations](https://mlsanity.com/paper/2303.01664/) (2023)
- [SESQA: semi-supervised learning for speech quality assessment](https://mlsanity.com/paper/2010.00368/) (2020)
- [WARP-Q: Quality Prediction For Generative Neural Speech Codecs](https://mlsanity.com/paper/2102.10449/) (2021)

## Built on by (most cited first, 24 of 24)

- [MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer](https://mlsanity.com/paper/2409.00750/) (2024)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement](https://mlsanity.com/paper/2501.15417/) (2025)
- [LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement](https://mlsanity.com/paper/2503.00493/) (2025)
- [FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching](https://mlsanity.com/paper/2505.19476/) (2025)
- [TSELM: Target Speaker Extraction using Discrete Tokens and Language Models](https://mlsanity.com/paper/2409.07841/) (2024)
- [Speech Enhancement Using Continuous Embeddings of Neural Audio Codec](https://mlsanity.com/paper/2502.16240/) (2025)
- [Universal Speech Enhancement with Regression and Generative Mamba](https://mlsanity.com/paper/2505.21198/) (2025)
- [Multi-Metric Preference Alignment for Generative Speech Restoration](https://mlsanity.com/paper/2508.17229/) (2025)
- [DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformers](https://mlsanity.com/paper/2504.09381/) (2025)
- [SonicMaster: Towards Controllable All-in-One Music Restoration and Mastering](https://mlsanity.com/paper/2508.03448/) (2025)
- [Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility](https://mlsanity.com/paper/2409.09357/) (2024)
- [FlowSE-GRPO: Training Flow Matching Speech Enhancement via Online Reinforcement Learning](https://mlsanity.com/paper/2601.16483/) (2026)
- [VoiceBridge: General Speech Restoration with One-step Latent Bridge Models](https://mlsanity.com/paper/2509.25275/) (2025)
- [UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement](https://mlsanity.com/paper/2510.20441/) (2025)
- [Aligning Generative Speech Enhancement with Perceptual Feedback](https://mlsanity.com/paper/2507.09929/) (2025)
- [SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancement](https://mlsanity.com/paper/2509.24708/) (2025)
- [MambAttention: Mamba with Multi-Head Attention for Generalizable Single-Channel Speech Enhancement](https://mlsanity.com/paper/2507.00966/) (2025)
- [High-Fidelity Speech Enhancement via Discrete Audio Tokens](https://mlsanity.com/paper/2510.02187/) (2025)
- [Post-Training Speech Enhancement Language Models with Perceptual Rewards](https://mlsanity.com/paper/2606.21458/) (2026)
- [Absorbing Discrete Diffusion for Speech Enhancement](https://mlsanity.com/paper/2602.22417/) (2026)
- [MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates](https://mlsanity.com/paper/2603.23048/) (2026)
- [UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations](https://mlsanity.com/paper/2604.14606/) (2026)
- [Masked Autoregressive Speech Enhancement with Continuous Neural Audio Codec Representations](https://mlsanity.com/paper/2609.03940/) (2026)

## Abstract (arXiv)

Speech restoration aims at restoring high quality speech in the presence of a diverse set of distortions. Although several deep learning paradigms have been studied for this task, the power of the recently emerging language models has not been fully explored. In this paper, we propose MaskSR, a masked language model capable of restoring full-band 44.1 kHz speech jointly considering noise, reverb, clipping, and low bandwidth. MaskSR works with discrete acoustic tokens extracted using a pre-trained neural codec. During training, MaskSR is optimized to predict randomly masked tokens extracted from the high quality target speech, conditioned on the corrupted speech with various distortions. During inference, MaskSR reconstructs the target speech tokens with efficient iterative sampling. Extensive experiments show that MaskSR obtains competitive results on both the full-band speech restoration task and also on sub-tasks compared with a wide range of models.