mlsanityData to 14 Oct 2026

UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation

arXiv 2503.00733 ↗First posted 2 Mar 2025eess.AS cs.CL cs.SDcited by 5

Not on the map. Why?

Abstract (arXiv)

Pre-training and representation learning have been playing an increasingly important role in modern speech processing. Nevertheless, different applications have been relying on different foundation models, since predominant pre-training techniques are either designed for discriminative tasks or generative tasks. In this work, we make the first attempt at building a unified pre-training framework for both types of tasks in speech. We show that with the appropriate design choices for pre-training, one can jointly learn a representation encoder and generative audio decoder that can be applied to both types of tasks. We propose UniWav, an encoder-decoder framework designed to unify pre-training representation learning and generative tasks. On speech recognition, text-to-speech, and speech tokenization, UniWav achieves comparable performance to different existing foundation models, each trained on a specific task. Our findings suggest that a single general-purpose foundation model for speech can be built to replace different foundation models, reducing the overhead and cost of pre-training.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 61 of its 69 references that are in the map. First posted on arXiv 2 Mar 2025.

Builds on

61 of 69 references.

References · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2022Flow Matching for Generative Modelingcited by 5,702
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
38 more
  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  3. 2022High Fidelity Neural Audio Compressioncited by 687
  4. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  5. 2020Jukebox: A Generative Model for Musiccited by 542
  6. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  7. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  8. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  9. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  10. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  11. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  13. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 309
  14. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  15. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  16. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  17. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 227
  18. 2023Listen, Think, and Understandcited by 209
  19. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  20. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 181
  21. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  22. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  23. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  24. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 145
  25. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 130
  26. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  27. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  28. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  29. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  30. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 91
  31. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  32. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 79
  33. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  34. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  35. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
  36. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  37. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  38. 2020Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesiscited by 54

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  2. 2025Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modelingcited by 7
  3. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Trainingcited by 3
  4. 2026Alethia: A Foundational Encoder for Voice Deepfakescited by 1
  5. 2026OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSLcited by 0