# Improving Speech Enhancement with Multi-Metric Supervision from Learned Quality Assessment

arXiv: https://arxiv.org/abs/2506.12260 · HTML: https://mlsanity.com/paper/2506.12260/

## Key facts

- Cited by 7 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 35 of its 54 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 30 of 35)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech](https://mlsanity.com/paper/2409.09305/) (2024)
- [Generalization Ability of MOS Prediction Networks](https://mlsanity.com/paper/2110.02635/) (2021)
- [OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer](https://mlsanity.com/paper/2401.16658/) (2024)
- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [The VoiceMOS Challenge 2022](https://mlsanity.com/paper/2203.11389/) (2022)
- [SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics](https://mlsanity.com/paper/2401.16812/) (2024)
- [How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASR](https://mlsanity.com/paper/2201.06685/) (2022)
- [SCOREQ: Speech Quality Assessment with Contrastive Regression](https://mlsanity.com/paper/2410.06675/) (2024)
- [URGENT Challenge: Universality, Robustness, and Generalizability For Speech Enhancement](https://mlsanity.com/paper/2406.04660/) (2024)
- [VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music](https://mlsanity.com/paper/2412.17667/) (2024)
- [ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models](https://mlsanity.com/paper/2401.17230/) (2024)
- [Bridging the Gap Between Monaural Speech Enhancement and Recognition with Distortion-Independent Acoustic Modeling](https://mlsanity.com/paper/1903.04567/) (2019)
- [ICASSP 2024 Speech Signal Improvement Challenge](https://mlsanity.com/paper/2401.14444/) (2024)
- [Interspeech 2025 URGENT Speech Enhancement Challenge](https://mlsanity.com/paper/2505.23212/) (2025)
- [Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment](https://mlsanity.com/paper/2502.05356/) (2025)
- [The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction](https://mlsanity.com/paper/2409.07001/) (2024)
- [Objective Measures of Perceptual Audio Quality Reviewed: An Evaluation of Their Application Domain Dependence](https://mlsanity.com/paper/2110.11438/) (2021)
- [MOS-Bench: Benchmarking Generalization Abilities of Subjective Speech Quality Assessment Models](https://mlsanity.com/paper/2411.03715/) (2024)
- [Closing the Gap Between Time-Domain Multi-Channel Speech Enhancement on Real and Simulation Conditions](https://mlsanity.com/paper/2110.14139/) (2021)
- [The CHiME-7 UDASE task: Unsupervised domain adaptation for conversational speech enhancement](https://mlsanity.com/paper/2307.03533/) (2023)

## Built on by (most cited first, 7 of 7)

- [UrgentMOS: Unified Multi-Metric and Preference Learning for Robust Speech Quality Assessment](https://mlsanity.com/paper/2601.18438/) (2026)
- [IntMeanFlow: Few-step Speech Generation with Integral Velocity Distillation](https://mlsanity.com/paper/2510.07979/) (2025)
- [URGENT-PK: Perceptually-Aligned Ranking Model Designed for Speech Enhancement Competition](https://mlsanity.com/paper/2506.23874/) (2025)
- [A Hybrid Discriminative and Generative System for Universal Speech Enhancement](https://mlsanity.com/paper/2601.19113/) (2026)
- [Improving Predicted MOS Scores, Not Perceived Quality: Multi-Predictor Test-Time Optimization of Enhanced Speech](https://mlsanity.com/paper/2609.39028/) (2026)
- [How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?](https://mlsanity.com/paper/2609.39032/) (2026)
- [Rethinking Batch Composition: Promoting Feature Diversity for Speech Enhancement](https://mlsanity.com/paper/2610.21803/) (2026)

## Abstract (arXiv)

Speech quality assessment (SQA) aims to predict the perceived quality of speech signals under a wide range of distortions. It is inherently connected to speech enhancement (SE), which seeks to improve speech quality by removing unwanted signal components. While SQA models are widely used to evaluate SE performance, their potential to guide SE training remains underexplored. In this work, we investigate a training framework that leverages a SQA model, trained to predict multiple evaluation metrics from a public SE leaderboard, as a supervisory signal for SE. This approach addresses a key limitation of conventional SE objectives, such as SI-SNR, which often fail to align with perceptual quality and generalize poorly across evaluation metrics. Moreover, it enables training on real-world data where clean references are unavailable. Experiments on both simulated and real-world test sets show that SQA-guided training consistently improves performance across a range of quality metrics. Code and checkpoints are available at https://github.com/urgent-challenge/urgent2026_challenge_track2