# Non-Intrusive Speech Intelligibility Prediction for Hearing Aids using Whisper and Metadata

arXiv: https://arxiv.org/abs/2309.09548 · HTML: https://mlsanity.com/paper/2309.09548/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [Generalization Ability of MOS Prediction Networks](https://mlsanity.com/paper/2110.02635/) (2021)
- [MBNet: MOS Prediction for Synthesized Speech with Mean-Bias Network](https://mlsanity.com/paper/2103.00110/) (2021)
- [Deep Learning-based Non-Intrusive Multi-Objective Speech Assessment Model with Cross-Domain Features](https://mlsanity.com/paper/2111.02363/) (2021)
- [ConferencingSpeech 2022 Challenge: Non-intrusive Objective Speech Quality Assessment (NISQA) Challenge for Online Conferencing Applications](https://mlsanity.com/paper/2203.16032/) (2022)
- [MBI-Net: A Non-Intrusive Multi-Branched Speech Intelligibility Prediction Model for Hearing Aids](https://mlsanity.com/paper/2204.03305/) (2022)
- [Fusion of Self-supervised Learned Models for MOS Prediction](https://mlsanity.com/paper/2204.04855/) (2022)
- [MTI-Net: A Multi-Target Speech Intelligibility Prediction Model](https://mlsanity.com/paper/2204.03310/) (2022)
- [Exploiting Hidden Representations from a DNN-based Speech Recogniser for Speech Intelligibility Prediction in Hearing-impaired Listeners](https://mlsanity.com/paper/2204.04287/) (2022)
- [The Effect of Spoken Language on Speech Enhancement using Self-Supervised Speech Representation Loss Functions](https://mlsanity.com/paper/2307.14502/) (2023)
- [Speech intelligibility of simulated hearing loss sounds and its prediction using the Gammachirp Envelope Similarity Index (GESI)](https://mlsanity.com/paper/2206.06573/) (2022)
- [Multi-Task Pseudo-Label Learning for Non-Intrusive Speech Quality Assessment Model](https://mlsanity.com/paper/2308.09262/) (2023)

## Built on by (most cited first, 4 of 4)

- [Transfer Learning from Whisper for Microscopic Intelligibility Prediction](https://mlsanity.com/paper/2404.01737/) (2024)
- [No Audiogram: Leveraging Existing Scores for Personalized Speech Intelligibility Prediction](https://mlsanity.com/paper/2506.02039/) (2025)
- [Non-Intrusive Binaural Speech Intelligibility Prediction Using Mamba for Hearing-Impaired Listeners](https://mlsanity.com/paper/2507.05729/) (2025)
- [Bottleneck Transformer-Based Approach for Improved Automatic STOI Score Prediction](https://mlsanity.com/paper/2602.15484/) (2026)

## Abstract (arXiv)

Automated speech intelligibility assessment is pivotal for hearing aid (HA) development. In this paper, we present three novel methods to improve intelligibility prediction accuracy and introduce MBI-Net+, an enhanced version of MBI-Net, the top-performing system in the 1st Clarity Prediction Challenge. MBI-Net+ leverages Whisper's embeddings to create cross-domain acoustic features and includes metadata from speech signals by using a classifier that distinguishes different enhancement methods. Furthermore, MBI-Net+ integrates the hearing-aid speech perception index (HASPI) as a supplementary metric into the objective function to further boost prediction performance. Experimental results demonstrate that MBI-Net+ surpasses several intrusive baseline systems and MBI-Net on the Clarity Prediction Challenge 2023 dataset, validating the effectiveness of incorporating Whisper embeddings, speech metadata, and related complementary metrics to improve prediction performance for HA.