mlsanityData to 7 Oct 2026

Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations

arXiv 2303.01664 ↗First posted 3 Mar 2023cs.SD cs.LG eess.AScited by 22

Not on the map. Why?

Abstract (arXiv)

Speech restoration (SR) is a task of converting degraded speech signals into high-quality ones. In this study, we propose a robust SR model called Miipher, and apply Miipher to a new SR application: increasing the amount of high-quality training data for speech generation by converting speech samples collected from the Web to studio-quality. To make our SR model robust against various degradation, we use (i) a speech representation extracted from w2v-BERT for the input feature, and (ii) a text representation extracted from transcripts via PnG-BERT as a linguistic conditioning feature. Experiments show that Miipher (i) is robust against various audio degradation and (ii) enable us to train a high-quality text-to-speech (TTS) model from restored speech samples collected from the Web. Audio samples are available at our demo page: google.github.io/df-conformer/miipher/

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 3 Mar 2023.

Builds on

40 of 53 references.

References · 40

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  9. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  11. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  12. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
28 more
  1. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  2. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  3. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  4. 2018Efficient Neural Audio Synthesiscited by 218
  5. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 213
  6. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  7. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  8. 2018Hierarchical Generative Modeling for Controllable Speech Synthesiscited by 93
  9. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  10. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  11. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  12. 2021PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTScited by 52
  13. 2021VoiceFixer: Toward General Speech Restoration with Neural Vocodercited by 50
  14. 2018Sample Efficient Adaptive Text-to-Speechcited by 45
  15. 2022SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shapingcited by 37
  16. 2017Audio Super Resolution using Neural Networkscited by 26
  17. 2019Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separationcited by 22
  18. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
  19. 2022SpeechPainter: Text-conditioned Speech Inpaintingcited by 17
  20. 2020A Curated Dataset of Urban Scenes for Audio-Visual Scene Analysiscited by 16
  21. 2021DF-Conformer: Integrated architecture of Conv-TasNet and Conformer using linear complexity self-attention for speech enhancementcited by 16
  22. 2021Dr-Vectors: Decision Residual Networks and an Improved Loss for Speaker Recognitioncited by 10
  23. 2022SQuId: Measuring Speech Naturalness in Many Languagescited by 10
  24. 2019Parametric Resynthesis with neural vocoderscited by 8
  25. 2020A survey and an extensive evaluation of popular audio declipping methodscited by 7
  26. 2022Attentive Temporal Pooling for Conformer-based Streaming Language Identification in Long-form Speechcited by 6
  27. 2022SelfRemaster: Self-Supervised Speech Restoration with Analysis-by-Synthesis Approach Using Channel Modelingcited by 4
  28. 2022Parameter-Free Attentive Scoring for Speaker Verificationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 12; 2026: 2.

22 papers.

Sort:

By area:

  1. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 66
  2. 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 25
  3. 2024MaskSR: Masked Language Model for Full-band Speech Restorationcited by 24
  4. 2024Controlling Emotion in Text-to-Speech with Natural Language Promptscited by 8
  5. 2024High-Resolution Speech Restoration with Latent Diffusion Modelcited by 8
  6. 2025Source Separation by Flow Matchingcited by 8
  7. 2024FINALLY: fast and universal speech enhancement with studio-like qualitycited by 7
  8. 2025Universal Speech Enhancement with Regression and Generative Mambacited by 7
  9. 2023The IMS Toucan System for the Blizzard Challenge 2023cited by 6
  10. 2025DiTSE: High-Fidelity Generative Speech Enhancement via Latent Diffusion Transformerscited by 6
  11. 2024Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibilitycited by 5
  12. 2025VoiceBridge: General Speech Restoration with One-step Latent Bridge Modelscited by 4
10 more
  1. 2025Linguistic Knowledge Transfer Learning for Speech Enhancementcited by 3
  2. 2025Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioningcited by 2
  3. 2025UniCoM: A Universal Code-Switching Speech Generatorcited by 2
  4. 2025AUDDT: A Unified Benchmark Toolkit for Audio and Speech Deepfake Detectorscited by 2
  5. 2025Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restorationcited by 1
  6. 2025Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesiscited by 1
  7. 2025ARTI-6: Towards Six-dimensional Articulatory Speech Encodingcited by 1
  8. 2025SpeechOp: Inference-Time Task Composition for Generative Speech Processingcited by 0
  9. 2026UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinationscited by 0
  10. 2026ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Modelscited by 0