mlsanityData to 7 Oct 2026

JOOCI: a Framework for Learning Comprehensive Speech Representations

arXiv 2410.11086 ↗First posted 14 Oct 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Information in speech can be categorized into two groups: Content (what is being said, such as linguistics) and Other (how it is expressed such as information about speaker and paralinguistic features). Current self-supervised learning (SSL) methods are shown to divide the model's representational-depth or layers in two, with earlier layers specializing in Other and later layers in Content related tasks. This layer-wise division is inherently sub-optimal, as neither information type can use all layers to build hierarchical representations. To address this, we propose JOOCI, a novel speech representation learning method that does not compromise on the representational-depth for either information type. JOOCI outperforms WavLM by 26.5%, and other models of similar size (100M parameters), when evaluated on two speaker recognition and two language tasks from the SUPERB benchmark, demonstrating its effectiveness in Jointly Optimizing Other and Content Information (JOOCI).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 44 of its 48 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

44 of 48 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
32 more
  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  2. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  3. 2022High Fidelity Neural Audio Compressioncited by 673
  4. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  5. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  6. 2019Res2Net: A New Multi-scale Backbone Architecturecited by 468
  7. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  8. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  9. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  10. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  11. 2018Additive Margin Softmax for Face Verificationcited by 275
  12. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  13. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  14. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  15. 2018Attentive Statistics Pooling for Deep Speaker Embeddingcited by 108
  16. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  17. 2020Unsupervised pretraining transfers well across languagescited by 100
  18. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 73
  19. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 43
  20. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  21. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  22. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 21
  23. 2023Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Predictioncited by 16
  24. 2021Self-Supervised Speaker Verification with Simple Siamese Network and Self-Supervised Regularizationcited by 14
  25. 2022CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Modelscited by 13
  26. 2022Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Datacited by 12
  27. 2022Pushing the limits of self-supervised speaker verification using regularized distillation frameworkcited by 11
  28. 2022Content-Context Factorized Representations for Automated Speech Recognitioncited by 8
  29. 2022Silence is Sweeter Than Speech: Self-Supervised Model Using Silence to Store Speaker Informationcited by 7
  30. 2023Exploration on HuBERT with Multiple Resolutionscited by 7
  31. 2023Analysing the Masked predictive coding training criterion for pre-training a Speech Representation Modelcited by 4
  32. 2024MS-HuBERT: Mitigating Pre-training and Inference Mismatch in Masked Language Modelling methods for learning Speech Representationscited by 2

Built on by

Not cited yet.