# P.808 Multilingual Speech Enhancement Testing: Approach and Results of URGENT 2025 Challenge

arXiv: https://arxiv.org/abs/2507.11306 · HTML: https://mlsanity.com/paper/2507.11306/

## Key facts

- Cited by 2 papers in the map (data through 19 Oct 2026).
- Placed in Speech enhancement and noise suppression, one of 11 directions in Speech and audio language models.
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 15 Jul 2025.

## Placed in

- Speech and audio language models / [Speech enhancement and noise suppression](https://mlsanity.com/speech/speech-enhancement-and-noise-suppression/)

## Builds on (most cited first, 26 of 26)

- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario](https://mlsanity.com/paper/2104.03603/) (2021)
- [ICASSP 2023 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2303.11510/) (2023)
- [EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation](https://mlsanity.com/paper/2406.06185/) (2024)
- [YODAS: Youtube-Oriented Dataset for Audio and Speech](https://mlsanity.com/paper/2406.00899/) (2024)
- [An Open source Implementation of ITU-T Recommendation P.808 with Validation](https://mlsanity.com/paper/2005.08138/) (2020)
- [URGENT Challenge: Universality, Robustness, and Generalizability For Speech Enhancement](https://mlsanity.com/paper/2406.04660/) (2024)
- [ICASSP 2024 Speech Signal Improvement Challenge](https://mlsanity.com/paper/2401.14444/) (2024)
- [Interspeech 2025 URGENT Speech Enhancement Challenge](https://mlsanity.com/paper/2505.23212/) (2025)
- [JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and Nonverbal Expressions](https://mlsanity.com/paper/2310.06072/) (2023)
- [On the Behavior of Intrusive and Non-intrusive Speech Enhancement Metrics in Predictive and Generative Settings](https://mlsanity.com/paper/2306.03014/) (2023)
- [Vibravox: A Dataset of French Speech Captured with Body-conduction Audio Sensors](https://mlsanity.com/paper/2407.11828/) (2024)
- [NESC: Robust Neural End-2-End Speech Coding with GANs](https://mlsanity.com/paper/2207.03282/) (2022)
- [Crowdsourced Multilingual Speech Intelligibility Testing](https://mlsanity.com/paper/2403.14817/) (2024)
- [Disentangling the Impacts of Language and Channel Variability on Speech Separation Networks](https://mlsanity.com/paper/2203.16040/) (2022)
- [Hallucination in Perceptual Metric-Driven Speech Enhancement Networks](https://mlsanity.com/paper/2403.11732/) (2024)
- [Application of Just-Noticeable Difference in Quality as Environment Suitability Test for Crowdsourcing Speech Quality Assessment Task](https://mlsanity.com/paper/2004.05502/) (2020)
- [Study on the Correlation between Objective Evaluations and Subjective Speech Quality and Intelligibility](https://mlsanity.com/paper/2307.04517/) (2023)
- [Language Barriers: Evaluating Cross-Lingual Performance of CNN and Transformer Architectures for Speech Quality Estimation](https://mlsanity.com/paper/2502.13004/) (2025)

## Built on by (most cited first, 2 of 2)

- [UrgentMOS: Unified Multi-Metric and Preference Learning for Robust Speech Quality Assessment](https://mlsanity.com/paper/2601.18438/) (2026)
- [DiffVQE: Hybrid Diffusion Voice Quality Enhancement Under Acoustic Echo and Noise](https://mlsanity.com/paper/2605.08189/) (2026)

## Abstract (arXiv)

In speech quality estimation for speech enhancement (SE) systems, subjective listening tests so far are considered as the gold standard. This should be even more true considering the large influx of new generative or hybrid methods into the field, revealing issues of some objective metrics. Efforts such as the Interspeech 2025 URGENT Speech Enhancement Challenge also involving non-English datasets add the aspect of multilinguality to the testing procedure. In this paper, we provide a brief recap of the ITU-T P.808 crowdsourced subjective listening test method. A first novel contribution is our proposed process of localizing both text and audio components of Naderi and Cutler's implementation of crowdsourced subjective absolute category rating (ACR) listening tests involving text-to-speech (TTS). Further, we provide surprising analyses of and insights into URGENT Challenge results, tackling the reliability of (P.808) ACR subjective testing as gold standard in the age of generative AI. Particularly, it seems that for generative SE methods, subjective (ACR MOS) and objective (DNSMOS, NISQA) reference-free metrics should be accompanied by objective phone fidelity metrics to reliably detect hallucinations. Finally, we will soon release our localization scripts and methods for easy deployment for new multilingual speech enhancement subjective evaluations according to ITU-T P.808.