mlsanityData to 7 Oct 2026

Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speech

arXiv 2303.11131 ↗First posted 20 Mar 2023cs.CL cs.LG cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning leverages unlabeled data effectively, improving label efficiency and generalization to domains without labeled data. While recent work has studied generalization to more acoustic/linguistic domains, languages, and modalities, these investigations are limited to single-source speech with one primary speaker in the recording. This paper presents Cocktail HuBERT, a self-supervised learning framework that generalizes to mixture speech using a masked pseudo source separation objective. This objective encourages the model to identify the number of sources, separate and understand the context, and infer the content of masked regions represented as discovered units. Cocktail HuBERT outperforms state-of-the-art results with 69% lower WER on multi-speaker ASR, 31% lower DER on diarization, and is competitive on single- and multi-speaker tasks from SUPERB.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

29 of 35 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  8. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  11. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  12. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
17 more
  1. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  2. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  3. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  4. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  5. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  6. 2021Unsupervised Speech Recognitioncited by 147
  7. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  8. 2020Learning Robust and Multilingual Speech Representationscited by 39
  9. 2018End-to-End Monaural Multi-speaker ASR System without Pretrainingcited by 26
  10. 2022SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Taskscited by 19
  11. 2019Transfer Learning from Audio-Visual Grounding to Speech Recognitioncited by 16
  12. 2021SpeechNet: A Universal Modularized Model for Speech Processing Taskscited by 16
  13. 2022Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognitioncited by 14
  14. 2021Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chaincited by 12
  15. 2022Simple and Effective Unsupervised Speech Synthesiscited by 12
  16. 2021Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problemcited by 11
  17. 2022Extended Graph Temporal Classification for Multi-Speaker End-to-End ASRcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 2.

6 papers.

  1. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  2. 2024Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognitioncited by 7
  3. 2025Serialized Output Prompting for Large Language Model-based Multi-Talker Speech Recognitioncited by 5
  4. 2023Spatial HuBERT: Self-supervised Spatial Speech Representation Learning for a Single Talker from Multi-channel Audiocited by 2
  5. 2024Speech Separation with Pretrained Frontend to Minimize Domain Mismatchcited by 1
  6. 2025Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separationcited by 0