mlsanityData to 7 Oct 2026

Towards Early Prediction of Self-Supervised Speech Model Performance

arXiv 2501.05966 ↗First posted 10 Jan 2025cs.SD cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In Self-Supervised Learning (SSL), pre-training and evaluation are resource intensive. In the speech domain, current indicators of the quality of SSL models during pre-training, such as the loss, do not correlate well with downstream performance. Consequently, it is often difficult to gauge the final downstream performance in a cost efficient manner during pre-training. In this work, we propose unsupervised efficient methods that give insights into the quality of the pre-training of SSL speech models, namely, measuring the cluster quality and rank of the embeddings of the SSL model. Results show that measures of cluster quality and rank correlate better with downstream performance than the pre-training loss with only one hour of unlabeled audio, reducing the need for GPU hours and labeled data in SSL model evaluation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

19 of 29 references.

References · 19

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  8. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  9. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  10. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  11. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  12. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
7 more
  1. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  2. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  3. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  4. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  5. 2024Open Implementation and Study of BEST-RQ for Speech Processingcited by 9
  6. 2024Towards Automatic Assessment of Self-Supervised Speech Models using Rankcited by 5
  7. 2024An Analysis of Linear Complexity Attention Substitutes with BEST-RQcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Modelscited by 2
  2. 2025Unify Variables in Neural Scaling Laws for General Audio Representations via Embedding Effective Rankcited by 0
  3. 2026Polynomial Mixing for Efficient Self-supervised Speech Encoderscited by 0
  4. 2026What Makes a Good Layer? Assessing the Layer-Wise Intrinsic Properties of Music Foundation Modelscited by 0
  5. 2026LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representationscited by 0