mlsanityData to 7 Oct 2026

UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data

arXiv 2101.07597 ↗First posted 19 Jan 2021cs.CL cs.LG cs.SDcited by 76

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a unified pre-training approach called UniSpeech to learn speech representations with both unlabeled and labeled data, in which supervised phonetic CTC learning and phonetically-aware contrastive self-supervised learning are conducted in a multi-task learning manner. The resultant representations can capture information more correlated with phonetic structures and improve the generalization across languages and domains. We evaluate the effectiveness of UniSpeech for cross-lingual representation learning on public CommonVoice corpus. The results show that UniSpeech outperforms self-supervised pretraining and supervised transfer learning for speech recognition by a maximum of 13.4% and 17.8% relative phone error rate reductions respectively (averaged over all testing languages). The transferability of UniSpeech is also demonstrated on a domain-shift speech recognition task, i.e., a relative word error rate reduction of 6% against the previous approach.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 19 Jan 2021.

Builds on

29 of 46 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  9. 2015Listen, Attend and Spellcited by 542
  10. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  11. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  12. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
17 more
  1. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  2. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  3. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  4. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  5. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  6. 2020Unsupervised pretraining transfers well across languagescited by 100
  7. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  8. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  9. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  10. 2016Achieving Human Parity in Conversational Speech Recognitioncited by 63
  11. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  12. 2020Learning Robust and Multilingual Speech Representationscited by 39
  13. 2017Transfer Learning for Speech Recognition on a Budgetcited by 25
  14. 2020Joint Masked CPC and CTC Training for ASRcited by 20
  15. 2020Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applicationscited by 11
  16. 2019BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognitioncited by 10
  17. 2020Transfer Learning Approaches for Streaming End-to-End Speech Recognition Systemcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 19; 2023: 15; 2024: 11; 2025: 8; 2026: 5.

76 papers.

Sort:

By area:

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  2. 2022AudioGen: Textually Guided Audio Generationcited by 292
  3. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  4. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  5. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  6. 2022Audio Self-supervised Learning: A Surveycited by 65
  7. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  8. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  9. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 50
  10. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  11. 2021End-to-end Speech Translation via Cross-modal Progressive Trainingcited by 47
  12. 2021Towards Building ASR Systems for the Next Billion Userscited by 45
38 more
  1. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  2. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  3. 2023Transformers in Speech Processing: A Surveycited by 38
  4. 2022textless-lib: a Library for Textless Spoken Language Processingcited by 28
  5. 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
  6. 2022A Survey of Multilingual Models for Automatic Speech Recognitioncited by 21
  7. 2023Finding Order in Chaos: A Novel Data Augmentation Method for Time Series in Contrastive Learningcited by 18
  8. 2024Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervisioncited by 17
  9. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognitioncited by 16
  10. 2021Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognitioncited by 15
  11. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15
  12. 2021Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstructioncited by 12
  13. 2022Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Datacited by 12
  14. 2024Unified Static and Dynamic Network: Efficient Temporal Filtering for Video Groundingcited by 12
  15. 2021XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognitioncited by 11
  16. 2023Adapting Multi-Lingual ASR Models for Handling Multiple Talkerscited by 11
  17. 2022Improving Mispronunciation Detection with Wav2vec2-based Momentum Pseudo-Labeling for Accentedness and Intelligibility Assessmentcited by 10
  18. 2022Boosting Cross-Domain Speech Recognition with Self-Supervisioncited by 10
  19. 2022A Brief Overview of Unsupervised Neural Speech Representation Learningcited by 9
  20. 2023Online Continual Learning in Keyword Spotting for Low-Resource Devices via Pooling High-Order Temporal Statisticscited by 9
  21. 2022MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targetscited by 8
  22. 2025Towards Interactive Intelligence for Digital Humanscited by 8
  23. 2021Pretext Tasks selection for multitask self-supervised speech representation learningcited by 7
  24. 2021Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASRcited by 7
  25. 2025Towards a Unified Representation Evaluation Framework Beyond Downstream Taskscited by 7
  26. 2023Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translationcited by 5
  27. 2026ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRAcited by 5
  28. 2021Multilingual Speech Recognition using Knowledge Transfer across Learning Processescited by 4
  29. 2022Efficient Utilization of Large Pre-Trained Models for Low Resource ASRcited by 4
  30. 2023Label Aware Speech Representation Learning For Language Identificationcited by 4
  31. 2023SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognitioncited by 4
  32. 2024A Backdoor Approach with Inverted Labels Using Dirty Label-Flipping Attackscited by 4
  33. 2021Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Maskcited by 3
  34. 2022Similarity and Content-based Phonetic Self Attention for Speech Recognitioncited by 3
  35. 2023Exploring Emotion Expression Recognition in Older Adults Interacting with a Virtual Coachcited by 3
  36. 2024The last Dance : Robust backdoor attack via diffusion models and bayesian approachcited by 3
  37. 2024Trading Devil: Robust backdoor attack via Stochastic investment models and Bayesian approachcited by 3
  38. 2024HebDB: a Weakly Supervised Dataset for Hebrew Speech Processingcited by 3

50 most cited of 76 All 76 (JSON)