mlsanityData to 14 Oct 2026

Investigating Pre-trained Audio Encoders in the Low-Resource Condition

arXiv 2305.17733 ↗First posted 28 May 2023cs.CL cs.SD eess.AScited by 11

Not on the map. Why?

Abstract (arXiv)

Pre-trained speech encoders have been central to pushing state-of-the-art results across various speech understanding and generation tasks. Nonetheless, the capabilities of these encoders in low-resource settings are yet to be thoroughly explored. To address this, we conduct a comprehensive set of experiments using a representative set of 3 state-of-the-art encoders (Wav2vec2, WavLM, Whisper) in the low-resource setting across 7 speech understanding and generation tasks. We provide various quantitative and qualitative analyses on task performance, convergence speed, and representational properties of the encoders. We observe a connection between the pre-training protocols of these encoders and the way in which they capture information in their internal layers. In particular, we observe the Whisper encoder exhibits the greatest low-resource capabilities on content-driven tasks in terms of performance and convergence speed.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 19 of its 22 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

19 of 22 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  9. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  10. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  11. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  12. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
7 more
  1. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  2. 2022Audio Self-supervised Learning: A Surveycited by 65
  3. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  4. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  5. 2020Similarity Analysis of Self-Supervised Speech Representationscited by 26
  6. 2022M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translationcited by 15
  7. 2022Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 3; 2026: 3.

11 papers.

  1. 2024Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Modelscited by 37
  2. 2023Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformerscited by 13
  3. 2026Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadoxcited by 7
  4. 2024Improving Domain-Specific ASR with LLM-Generated Contextual Descriptionscited by 6
  5. 2025WhiSPA: Semantically and Psychologically Aligned Whisper with Self-Supervised Contrastive and Student-Teacher Learningcited by 3
  6. 2024Zero-Shot End-To-End Spoken Question Answering In Medical Domaincited by 2
  7. 2026Resurfacing Paralinguistic Awareness in Large Audio Language Modelscited by 2
  8. 2024Double Mixture: Towards Continual Event Detection from Speechcited by 1
  9. 2025Continual Speech Learning with Fused Speech Featurescited by 1
  10. 2025An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-trainingcited by 1
  11. 2026Reusing Latent Speech Representations for Query-Conditioned Topic Localization in Transcriptscited by 0