mlsanityData to 14 Oct 2026

Multi-task self-supervised learning for Robust Speech Recognition

arXiv 2001.09239 ↗First posted 25 Jan 2020eess.AS cs.CL cs.LGcited by 130

Not on the map. Why?

Abstract (arXiv)

Despite the growing interest in unsupervised learning, extracting meaningful knowledge from unlabelled audio remains an open challenge. To take a step in this direction, we recently proposed a problem-agnostic speech encoder (PASE), that combines a convolutional encoder followed by multiple neural networks, called workers, tasked to solve self-supervised problems (i.e., ones that do not require manual annotations as ground truth). PASE was shown to capture relevant speech information, including speaker voice-print and phonemes. This paper proposes PASE+, an improved version of PASE for robust speech recognition in noisy and reverberant environments. To this end, we employ an online speech distortion module, that contaminates the input signals with a variety of random disturbances. We then propose a revised encoder that better learns short- and long-term speech dynamics with an efficient combination of recurrent and convolutional networks. Finally, we refine the set of workers used in self-supervision to encourage better cooperation. Results on TIMIT, DIRHA and CHiME-5 show that PASE+ significantly outperforms both the previous version of PASE as well as common acoustic features. Interestingly, PASE+ learns transferable representations suitable for highly mismatched acoustic conditions.

Key facts. Cited by 130 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 46; 2022: 41; 2023: 21; 2024: 14; 2025: 3; 2026: 5.

130 papers.

Sort:

By area:

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  4. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  5. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 312
  6. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  7. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  8. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  9. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  10. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  11. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  12. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
38 more
  1. 2021Contrastive Learning of Musical Representationscited by 82
  2. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  3. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  4. 2021UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Datacited by 77
  5. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 77
  6. 2022ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakerscited by 74
  7. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  8. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 66
  9. 2022Audio Self-supervised Learning: A Surveycited by 65
  10. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  11. 2023Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learningcited by 48
  12. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  13. 2021UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Trainingcited by 45
  14. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 44
  15. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
  16. 2022Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalizationcited by 35
  17. 2021Textless Speech Emotion Conversion using Discrete and Decomposed Representationscited by 29
  18. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  19. 2023Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022cited by 29
  20. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  21. 2022Detecting Dysfluencies in Stuttering Therapy Using wav2vec 2.0cited by 27
  22. 2021Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learningcited by 26
  23. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  24. 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learningcited by 25
  25. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  26. 2024Temporal-Channel Modeling in Multi-head Self-Attention for Synthetic Speech Detectioncited by 25
  27. 2022Self-Supervised Anomaly Detection in Computer Vision and Beyond: A Survey and Outlookcited by 24
  28. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 21
  29. 2021Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptationcited by 19
  30. 2021Multi-Task Self-Supervised Pre-Training for Music Classificationcited by 18
  31. 2022Adaptive Memory Networks with Self-supervised Learning for Unsupervised Anomaly Detectioncited by 18
  32. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 18
  33. 2022Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptationcited by 17
  34. 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 17
  35. 2021Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognitioncited by 16
  36. 2021Membership Inference Attacks Against Self-supervised Speech Modelscited by 16
  37. 2021Robust Optimization for Multilingual Translation with Imbalanced Datacited by 15
  38. 2021Self-Supervised Learning for speech recognition with Intermediate layer supervisioncited by 15

50 most cited of 130 All 130 (JSON)