mlsanityData to 7 Oct 2026

How to Estimate Model Transferability of Pre-Trained Speech Models?

arXiv 2306.01015 ↗First posted 1 Jun 2023cs.CL cs.NE cs.SDcited by 10

Not on the map. Why?

Abstract (arXiv)

In this work, we introduce a "score-based assessment" framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks. We leverage upon two representation theories, Bayesian likelihood estimation and optimal transport, to generate rank scores for the PSM candidates using the extracted representations. Our framework efficiently computes transferability scores without actual fine-tuning of candidate models or layers by making a temporal independent hypothesis. We evaluate some popular supervised speech models (e.g., Conformer RNN-Transducer) and self-supervised speech models (e.g., HuBERT) in cross-layer and cross-model settings using public data. Experimental results show a high Spearman's rank correlation and low p-value between our estimation framework and fine-tuning ground truth. Our proposed transferability framework requires less computational time and resources, making it a resource-saving and time-efficient approach for tuning speech foundation models.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

36 of 49 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  10. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  12. 2021AST: Audio Spectrogram Transformercited by 634
24 more
  1. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  2. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  3. 2019Generalized Sliced Wasserstein Distancescited by 205
  4. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 182
  5. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  6. 2020Geometric Dataset Distances via Optimal Transportcited by 157
  7. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  8. 2019Transferability and Hardness of Supervised Classification Taskscited by 125
  9. 2021Voice2Series: Reprogramming Acoustic Models for Time Series Classificationcited by 91
  10. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  11. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  12. 2022Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 31
  13. 2022How stable are Transferability Metrics evaluations?cited by 30
  14. 2021Towards More Fine-grained and Reliable NLP Performance Predictioncited by 26
  15. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  16. 2021Transferability Metrics for Selecting Source Model Ensemblescited by 24
  17. 2023From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognitioncited by 23
  18. 2021On Addressing Practical Challenges for RNN-Transducercited by 19
  19. 2021Scaling ASR Improves Zero and Few Shot Learningcited by 15
  20. 2019Audio tagging with noisy labels and minimal supervisioncited by 14
  21. 2020Device-Robust Acoustic Scene Classification Based on Two-Stage Categorization and Data Augmentationcited by 13
  22. 2022CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Modelscited by 13
  23. 2023Efficient Domain Adaptation for Speech Foundation Modelscited by 13
  24. 2022Evidence > Intuition: Transferability Estimation for Encoder Selectioncited by 7

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 3; 2025: 2.

10 papers.

  1. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  2. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
  3. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  4. 2023Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognitioncited by 13
  5. 2024Which Model to Transfer? A Survey on Transferability Estimationcited by 13
  6. 2023Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classificationcited by 10
  7. 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
  8. 2025Spoken Conversational Agents with Large Language Modelscited by 4
  9. 2025How NOT to benchmark your SITE metric: Beyond Static Leaderboards and Towards Realistic Evaluationcited by 2
  10. 2023Representation Learning With Hidden Unit Clustering For Low Resource Speech Applicationscited by 1