# Generative Speech Foundation Model Pretraining for High-Quality Speech Extraction and Restoration

arXiv: https://arxiv.org/abs/2409.16117 · HTML: https://mlsanity.com/paper/2409.16117/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2024.

## Built on by (most cited first, 16 of 16)

- [AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement](https://mlsanity.com/paper/2501.15417/) (2025)
- [LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement](https://mlsanity.com/paper/2503.00493/) (2025)
- [Audio Super-Resolution with Latent Bridge Models](https://mlsanity.com/paper/2509.17609/) (2025)
- [A2SB: Audio-to-Audio Schrodinger Bridges](https://mlsanity.com/paper/2501.11311/) (2025)
- [SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline](https://mlsanity.com/paper/2505.19314/) (2025)
- [Universal Speech Enhancement with Regression and Generative Mamba](https://mlsanity.com/paper/2505.21198/) (2025)
- [Multi-Metric Preference Alignment for Generative Speech Restoration](https://mlsanity.com/paper/2508.17229/) (2025)
- [VoiceBridge: General Speech Restoration with One-step Latent Bridge Models](https://mlsanity.com/paper/2509.25275/) (2025)
- [Towards Real-Time Generative Speech Restoration with Flow-Matching](https://mlsanity.com/paper/2510.16997/) (2025)
- [MeanFlow-TSE: One-Step Generative Target Speaker Extraction with Mean Flow](https://mlsanity.com/paper/2512.18572/) (2025)
- [Adaptive Deterministic Flow Matching for Target Speaker Extraction](https://mlsanity.com/paper/2510.16995/) (2025)
- [Unsupervised Single-Channel Audio Separation with Diffusion Source Priors](https://mlsanity.com/paper/2512.07226/) (2025)
- [Generative AI Meets Wireless Sensing: Towards Wireless Foundation Model](https://mlsanity.com/paper/2509.15258/) (2025)
- [AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow](https://mlsanity.com/paper/2603.10701/) (2026)
- [PhASE-Flow: Phonetic-Conditioned Acoustic Flow Matching in SSL Representation Domain for Speech Enhancement](https://mlsanity.com/paper/2606.17806/) (2026)
- [Exploring Efficient Waveform Diffusion Models for Foley Sound Generation](https://mlsanity.com/paper/2607.29148/) (2026)

## Abstract (arXiv)

This paper proposes a generative pretraining foundation model for high-quality speech restoration tasks. By directly operating on complex-valued short-time Fourier transform coefficients, our model does not rely on any vocoders for time-domain signal reconstruction. As a result, our model simplifies the synthesis process and removes the quality upper-bound introduced by any mel-spectrogram vocoder compared to prior work SpeechFlow. The proposed method is evaluated on multiple speech restoration tasks, including speech denoising, bandwidth extension, codec artifact removal, and target speaker extraction. In all scenarios, finetuning our pretrained model results in superior performance over strong baselines. Notably, in the target speaker extraction task, our model outperforms existing systems, including those leveraging SSL-pretrained encoders like WavLM. The code and the pretrained checkpoints are publicly available in the NVIDIA NeMo framework.