mlsanityData to 7 Oct 2026

Speech foundation models on intelligibility prediction for hearing-impaired listeners

arXiv 2401.14289 ↗First posted 24 Jan 2024cs.SD cs.LG eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

Speech foundation models (SFMs) have been benchmarked on many speech processing tasks, often achieving state-of-the-art performance with minimal adaptation. However, the SFM paradigm has been significantly less explored for applications of interest to the speech perception community. In this paper we present a systematic evaluation of 10 SFMs on one such application: Speech intelligibility prediction. We focus on the non-intrusive setup of the Clarity Prediction Challenge 2 (CPC2), where the task is to predict the percentage of words correctly perceived by hearing-impaired listeners from speech-in-noise recordings. We propose a simple method that learns a lightweight specialized prediction head on top of frozen SFMs to approach the problem. Our results reveal statistically significant differences in performance across SFMs. Our method resulted in the winning submission in the CPC2, demonstrating its promise for speech perception applications.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 14 of its 21 references that are in the map. First posted on arXiv 24 Jan 2024.

Builds on

14 of 21 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  10. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  11. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  12. 2022MBI-Net: A Non-Intrusive Multi-Branched Speech Intelligibility Prediction Model for Hearing Aidscited by 11
2 more
  1. 2023Non Intrusive Intelligibility Predictor for Hearing Impaired Individuals using Self Supervised Speech Representationscited by 3
  2. 2020Clarity: Machine Learning Challenges to Revolutionise Hearing Device Processingcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 5.

6 papers.

  1. 2025Unveiling the Best Practices for Applying Speech Foundation Models to Speech Intelligibility Prediction for Hearing-Impaired Peoplecited by 3
  2. 2024Transfer Learning from Whisper for Microscopic Intelligibility Predictioncited by 1
  3. 2025Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddingscited by 1
  4. 2025No Audiogram: Leveraging Existing Scores for Personalized Speech Intelligibility Predictioncited by 0
  5. 2025Non-Intrusive Binaural Speech Intelligibility Prediction Using Mamba for Hearing-Impaired Listenerscited by 0
  6. 2025Leveraging Multiple Speech Enhancers for Non-Intrusive Intelligibility Prediction for Hearing-Impaired Listenerscited by 0