mlsanityData to 7 Oct 2026

Generative Pre-training for Speech with Flow Matching

arXiv 2310.16338 ↗First posted 25 Oct 2023eess.AS cs.CL cs.LGcited by 55

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Generative models have gained more and more attention in recent years for their remarkable success in tasks that required estimating and sampling data distribution to generate high-fidelity synthetic data. In speech, text-to-speech synthesis and neural vocoder are good examples where generative models have shined. While generative models have been applied to different applications in speech, there exists no general-purpose generative model that models speech directly. In this work, we take a step toward this direction by showing a single pre-trained generative model can be adapted to different downstream tasks with strong performance. Specifically, we pre-trained a generative model, named SpeechFlow, on 60k hours of untranscribed speech with Flow Matching and masked conditions. Experiment results show the pre-trained generative model can be fine-tuned with task-specific data to match or surpass existing expert models on speech enhancement, separation, and synthesis. Our work suggested a foundational model for generation tasks in speech can be built with generative pre-training.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 59 of its 75 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

59 of 75 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Diffusion-based Generative Speech Source Separation31 Oct
  2. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Models11 Aug
  3. 2022SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping31 Mar
  4. 2022Conditional Diffusion Probabilistic Model for Speech Enhancement10 Feb
  5. 2021A Study on Speech Enhancement Based on Diffusion Probabilistic Model25 Jul

Other references · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2022Flow Matching for Generative Modelingcited by 5,426
  10. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  11. 2018Neural Ordinary Differential Equationscited by 3,733
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
38 more
  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  8. 2022High Fidelity Neural Audio Compressioncited by 673
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  10. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  11. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  12. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  13. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  14. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  15. 2018SDR - half-baked or well done?cited by 420
  16. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  17. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  18. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  19. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  20. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  21. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 277
  22. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  23. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  24. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  25. 2020Attention is All You Need in Speech Separationcited by 184
  26. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  27. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  28. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  29. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Resultscited by 128
  30. 2022Generative Spoken Dialogue Language Modelingcited by 125
  31. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  32. 2021MetricGAN+: An Improved Version of MetricGAN for Speech Enhancementcited by 114
  33. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  34. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  35. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  36. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  37. 2020Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesiscited by 54
  38. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 17; 2025: 30; 2026: 7.

55 papers.

Sort:

By area:

  1. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  2. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimizationcited by 67
  3. 2024Riemannian Flow Matching Policy for Robot Motion Learningcited by 61
  4. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  5. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generationcited by 36
  6. 2024Diffusion Models for Audio Restorationcited by 32
  7. 2024LAFMA: A Latent Flow Matching Model for Text-to-Audio Generationcited by 24
  8. 2025AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancementcited by 23
  9. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 22
  10. 2025FlowSE: Flow Matching-based Speech Enhancementcited by 20
  11. 2024Fast and Robust Visuomotor Riemannian Flow Matching Policycited by 18
  12. 2025FLowHigh: Towards Efficient and High-Quality Audio Super-Resolution with Single-Step Flow Matchingcited by 18
38 more
  1. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  2. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  3. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlowcited by 16
  4. 2025MaskFlow: Discrete Flows For Flexible and Efficient Long Video Generationcited by 12
  5. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  6. 2024RFWave: Multi-band Rectified Flow for Audio Waveform Reconstructioncited by 9
  7. 2024Noise-robust Speech Separation with Fast Generative Correctioncited by 9
  8. 2025Designing a Conditional Prior Distribution for Flow-Based Generative Modelscited by 9
  9. 2025From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speechcited by 9
  10. 2025SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipelinecited by 9
  11. 2025Universal Speech Enhancement with Regression and Generative Mambacited by 7
  12. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  13. 2025OAT-FM: Optimal Acceleration Transport for Improved Flow Matchingcited by 7
  14. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 6
  15. 2025UniWav: Towards Unified Pre-training for Speech Representation Learning and Generationcited by 5
  16. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Predictioncited by 5
  17. 2025Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modelingcited by 5
  18. 2024An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTScited by 4
  19. 2025VoiceBridge: General Speech Restoration with One-step Latent Bridge Modelscited by 4
  20. 2025Is There a Better Source Distribution than Gaussian? Exploring Source Distributions for Image Flow Matchingcited by 4
  21. 2025MeanFlow-TSE: One-Step Generative Target Speaker Extraction with Mean Flowcited by 4
  22. 2024A Closer Look at Neural Codec Resynthesis: Bridging the Gap between Codec and Waveform Generationcited by 3
  23. 2025VoicePrompter: Robust Zero-Shot Voice Conversion with Voice Prompt and Conditional Flow Matchingcited by 3
  24. 2025Towards Real-Time Generative Speech Restoration with Flow-Matchingcited by 3
  25. 2025Flow Matching in the Low-Noise Regime: Pathologies and a Contrastive Remedycited by 2
  26. 2025High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modelingcited by 2
  27. 2025Adaptive Deterministic Flow Matching for Target Speaker Extractioncited by 2
  28. 2026Geometry-Aware Image Flow Matchingcited by 2
  29. 2024Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuningcited by 1
  30. 2024Takin-VC: Expressive Zero-Shot Voice Conversion via Adaptive Hybrid Content Encoding and Enhanced Timbre Modelingcited by 1
  31. 2026Alethia: A Foundational Encoder for Voice Deepfakescited by 1
  32. 2024Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translationcited by 0
  33. 2024Gen-Swarms: Adapting Deep Generative Models to Swarms of Dronescited by 0
  34. 2025Real-Time Person Image Synthesis Using a Flow Matching Modelcited by 0
  35. 2025Decision Flow Policy Optimizationcited by 0
  36. 2025Speech Enhancement based on cascaded two flowscited by 0
  37. 2025SpeechOp: Inference-Time Task Composition for Generative Speech Processingcited by 0
  38. 2025LP-CFM: Perceptual Invariance-Aware Conditional Flow Matching for Speech Modelingcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching
  2. 2025UniFlow: Unifying Speech Front-End Tasks via Continuous Generative Modeling
  3. 2025MeanFlowSE: One-Step Generative Speech Enhancement via MeanFlow
  4. 2025Real-Time Streamable Generative Speech Restoration with Flow Matching
  5. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matching
  6. 2026Real-time Speech Restoration using Data Prediction Mean Flows
  7. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI
  8. 2025Shortcut Flow Matching for Speech Enhancement: Step-Invariant flows via single stage training