mlsanityData to 7 Oct 2026

Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing

arXiv 2211.01522 ↗First posted 2 Nov 2022cs.LG cs.SD eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) for rich speech representations has achieved empirical success in low-resource Automatic Speech Recognition (ASR) and other speech processing tasks, which can mitigate the necessity of a large amount of transcribed speech and thus has driven a growing demand for on-device ASR and other speech processing. However, advanced speech SSL models have become increasingly large, which contradicts the limited on-device resources. This gap could be more severe in multilingual/multitask scenarios requiring simultaneously recognizing multiple languages or executing multiple speech processing tasks. Additionally, strongly overparameterized speech SSL models tend to suffer from overfitting when being finetuned on low-resource speech corpus. This work aims to enhance the practical usage of speech SSL models towards a win-win in both enhanced efficiency and alleviated overfitting via our proposed S3-Router framework, which for the first time discovers that simply discarding no more than 10% of model weights via only finetuning model connections of speech SSL models can achieve better accuracy over standard weight finetuning on downstream speech processing tasks. More importantly, S3-Router can serve as an all-in-one technique to enable (1) a new finetuning scheme, (2) an efficient multilingual/multitask solution, (3) a state-of-the-art ASR pruning technique, and (4) a new tool to quantitatively analyze the learned speech representation. We believe S3-Router has provided a new perspective for practical deployment of speech SSL models. Our codes are available at: https://github.com/GATECH-EIC/S3-Router.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 79 of its 136 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

79 of 136 references.

References · 79

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
38 more
  1. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  2. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  3. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  4. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  6. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  8. 2018Rethinking the Value of Network Pruningcited by 595
  9. 2015Listen, Attend and Spellcited by 542
  10. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  11. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  12. 2015Attention-Based Models for Speech Recognitioncited by 400
  13. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  14. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  15. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  16. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  17. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  18. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  19. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  20. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  21. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  22. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  23. 2020Unsupervised pretraining transfers well across languagescited by 100
  24. 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 95
  25. 2017Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Datacited by 92
  26. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  27. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  28. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  29. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 86
  30. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  31. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  32. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  33. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  34. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  35. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  36. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  37. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  38. 2022Audio Self-supervised Learning: A Surveycited by 65

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 2.

7 papers.

  1. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 70
  2. 2023Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learningcited by 9
  3. 2023Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learningcited by 3
  4. 2023Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Modelcited by 3
  5. 2024A Parameter-efficient Language Extension Framework for Multilingual ASRcited by 3
  6. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  7. 2025Audio-3DVG: Unified Audio -- Point Cloud Fusion for 3D Visual Groundingcited by 0