# Using Speech Foundational Models in Loss Functions for Hearing Aid Speech Enhancement

arXiv: https://arxiv.org/abs/2407.13333 · HTML: https://mlsanity.com/paper/2407.13333/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aids](https://mlsanity.com/paper/2005.11138/) (2020)
- [A consolidated view of loss functions for supervised deep learning-based speech enhancement](https://mlsanity.com/paper/2009.12286/) (2020)
- [On End-to-end Multi-channel Time Domain Speech Separation in Reverberant Environments](https://mlsanity.com/paper/2011.05958/) (2020)
- [Perceive and predict: self-supervised speech representation based loss functions for speech enhancement](https://mlsanity.com/paper/2301.04388/) (2023)
- [Non-Intrusive Speech Intelligibility Prediction for Hearing-Impaired Users using Intermediate ASR Features and Human Memory Models](https://mlsanity.com/paper/2401.13611/) (2024)
- [Multi-Channel Target Speaker Extraction with Refinement: The WavLab Submission to the Second Clarity Enhancement Challenge](https://mlsanity.com/paper/2302.07928/) (2023)
- [Speech foundation models on intelligibility prediction for hearing-impaired listeners](https://mlsanity.com/paper/2401.14289/) (2024)
- [The Effect of Spoken Language on Speech Enhancement using Self-Supervised Speech Representation Loss Functions](https://mlsanity.com/paper/2307.14502/) (2023)
- [Restoring speech intelligibility for hearing aid users with deep learning](https://mlsanity.com/paper/2206.11567/) (2022)
- [Multi-CMGAN+/+: Leveraging Multi-Objective Speech Quality Metric Prediction for Speech Enhancement](https://mlsanity.com/paper/2312.08979/) (2023)
- [Non Intrusive Intelligibility Predictor for Hearing Impaired Individuals using Self Supervised Speech Representations](https://mlsanity.com/paper/2307.13423/) (2023)
- [Optimising Hearing Aid Fittings for Speech in Noise with a Differentiable Hearing Loss Model](https://mlsanity.com/paper/2106.04639/) (2021)

## Abstract (arXiv)

Machine learning techniques are an active area of research for speech enhancement for hearing aids, with one particular focus on improving the intelligibility of a noisy speech signal. Recent work has shown that feature encodings from self-supervised speech representation models can effectively capture speech intelligibility. In this work, it is shown that the distance between self-supervised speech representations of clean and noisy speech correlates more strongly with human intelligibility ratings than other signal-based metrics. Experiments show that training a speech enhancement model using this distance as part of a loss function improves the performance over using an SNR-based loss function, demonstrated by an increase in HASPI, STOI, PESQ and SI-SNR scores. This method takes inference of a high parameter count model only at training time, meaning the speech enhancement model can remain smaller, as is required for hearing aids.