mlsanityData to 14 Oct 2026

Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuning

arXiv 2110.09930 ↗First posted 18 Oct 2021eess.AS cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Speech representation learning plays a vital role in speech processing. Among them, self-supervised learning (SSL) has become an important research direction. It has been shown that an SSL pretraining model can achieve excellent performance in various downstream tasks of speech processing. On the other hand, supervised multi-task learning (MTL) is another representation learning paradigm, which has been proven effective in computer vision (CV) and natural language processing (NLP). However, there is no systematic research on the general representation learning model trained by supervised MTL in speech processing. In this paper, we show that MTL finetuning can further improve SSL pretraining. We analyze the generalizability of supervised MTL finetuning to examine if the speech representation learned by MTL finetuning can generalize to unseen new tasks.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

22 of 29 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  7. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  8. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  11. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
  12. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
10 more
  1. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  2. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  3. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  4. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  5. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
  6. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 130
  7. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  8. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  9. 2020Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretrainingcited by 35
  10. 2021SpeechNet: A Universal Modularized Model for Speech Processing Taskscited by 16

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 1.

4 papers.

  1. 2022Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processingcited by 7
  2. 2023MERTech: Instrument Playing Technique Detection Using Self-Supervised Pretrained Model With Multi-Task Finetuningcited by 4
  3. 2024TokenVerse: Towards Unifying Speech and NLP Tasks via Transducer-based ASRcited by 2
  4. 2025Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalizationcited by 2