# Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations

arXiv: https://arxiv.org/abs/2303.01664 · HTML: https://mlsanity.com/paper/2303.01664/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 53 references that are in the map.
- First posted on arXiv 3 Mar 2023.

## Builds on (most cited first, 30 of 40)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [Supervised Speech Separation Based on Deep Learning: An Overview](https://mlsanity.com/paper/1708.07524/) (2017)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network](https://mlsanity.com/paper/2104.02133/) (2021)
- [Universal Speech Enhancement with Score-based Diffusion](https://mlsanity.com/paper/2206.03065/) (2022)
- [Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling](https://mlsanity.com/paper/2010.04301/) (2020)
- [PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS](https://mlsanity.com/paper/2103.15060/) (2021)
- [VoiceFixer: Toward General Speech Restoration with Neural Vocoder](https://mlsanity.com/paper/2109.13731/) (2021)
- [Sample Efficient Adaptive Text-to-Speech](https://mlsanity.com/paper/1809.10460/) (2018)
- [SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping](https://mlsanity.com/paper/2203.16749/) (2022)
- [Audio Super Resolution using Neural Networks](https://mlsanity.com/paper/1708.00853/) (2017)
- [Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation](https://mlsanity.com/paper/1904.04169/) (2019)
- [WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration](https://mlsanity.com/paper/2210.01029/) (2022)

## Built on by (most cited first, 22 of 22)

- [Natural language guidance of high-fidelity text-to-speech with synthetic annotations](https://mlsanity.com/paper/2402.01912/) (2024)
- [PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions](https://mlsanity.com/paper/2309.08140/) (2023)
- [MaskSR: Masked Language Model for Full-band Speech Restoration](https://mlsanity.com/paper/2406.02092/) (2024)
- [Controlling Emotion in Text-to-Speech with Natural Language Prompts](https://mlsanity.com/paper/2406.06406/) (2024)
- [High-Resolution Speech Restoration with Latent Diffusion Model](https://mlsanity.com/paper/2409.11145/) (2024)
- [Source Separation by Flow Matching](https://mlsanity.com/paper/2505.16119/) (2025)
- [FINALLY: fast and universal speech enhancement with studio-like quality](https://mlsanity.com/paper/2410.05920/) (2024)
- [Universal Speech Enhancement with Regression and Generative Mamba](https://mlsanity.com/paper/2505.21198/) (2025)
- [The IMS Toucan System for the Blizzard Challenge 2023](https://mlsanity.com/paper/2310.17499/) (2023)
- [DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformers](https://mlsanity.com/paper/2504.09381/) (2025)
- [Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility](https://mlsanity.com/paper/2409.09357/) (2024)
- [VoiceBridge: General Speech Restoration with One-step Latent Bridge Models](https://mlsanity.com/paper/2509.25275/) (2025)
- [Linguistic Knowledge Transfer Learning for Speech Enhancement](https://mlsanity.com/paper/2503.07078/) (2025)
- [Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioning](https://mlsanity.com/paper/2506.04527/) (2025)
- [UniCoM: A Universal Code-Switching Speech Generator](https://mlsanity.com/paper/2508.15244/) (2025)
- [AUDDT: A Unified Benchmark Toolkit for Audio and Speech Deepfake Detectors](https://mlsanity.com/paper/2509.21597/) (2025)
- [Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restoration](https://mlsanity.com/paper/2505.04457/) (2025)
- [Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis](https://mlsanity.com/paper/2505.18972/) (2025)
- [ARTI-6: Towards Six-dimensional Articulatory Speech Encoding](https://mlsanity.com/paper/2509.21447/) (2025)
- [SpeechOp: Inference-Time Task Composition for Generative Speech Processing](https://mlsanity.com/paper/2509.14298/) (2025)
- [UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations](https://mlsanity.com/paper/2604.14606/) (2026)
- [ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Models](https://mlsanity.com/paper/2607.09134/) (2026)

## Abstract (arXiv)

Speech restoration (SR) is a task of converting degraded speech signals into high-quality ones. In this study, we propose a robust SR model called Miipher, and apply Miipher to a new SR application: increasing the amount of high-quality training data for speech generation by converting speech samples collected from the Web to studio-quality. To make our SR model robust against various degradation, we use (i) a speech representation extracted from w2v-BERT for the input feature, and (ii) a text representation extracted from transcripts via PnG-BERT as a linguistic conditioning feature. Experiments show that Miipher (i) is robust against various audio degradation and (ii) enable us to train a high-quality text-to-speech (TTS) model from restored speech samples collected from the Web. Audio samples are available at our demo page: google.github.io/df-conformer/miipher/