# An Open source Implementation of ITU-T Recommendation P.808 with Validation

arXiv: https://arxiv.org/abs/2005.08138 · HTML: https://mlsanity.com/paper/2005.08138/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2020.

## Built on by (most cited first, 27 of 27)

- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Speech Enhancement and Dereverberation with Diffusion-based Generative Models](https://mlsanity.com/paper/2208.05830/) (2022)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [Interspeech 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2101.01902/) (2021)
- [S3PRL-VC: Open-source Voice Conversion Framework with Self-supervised Speech Representations](https://mlsanity.com/paper/2110.06280/) (2021)
- [MetricGAN-U: Unsupervised speech enhancement/ dereverberation based only on noisy/ reverberated speech](https://mlsanity.com/paper/2110.05866/) (2021)
- [A Comparative Study of Self-supervised Speech Representation Based Voice Conversion](https://mlsanity.com/paper/2207.04356/) (2022)
- [Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/2402.12220/) (2024)
- [A Text-to-Speech Pipeline, Evaluation Methodology, and Initial Fine-Tuning Results for Child Speech Synthesis](https://mlsanity.com/paper/2203.11562/) (2022)
- [Unsupervised Data Selection for TTS: Using Arabic Broadcast News as a Case Study](https://mlsanity.com/paper/2301.09099/) (2023)
- [On Prosody Modeling for ASR+TTS based Voice Conversion](https://mlsanity.com/paper/2107.09477/) (2021)
- [A Closer Look at Neural Codec Resynthesis: Bridging the Gap between Codec and Waveform Generation](https://mlsanity.com/paper/2410.22448/) (2024)
- [Pre-training Feature Guided Diffusion Model for Speech Enhancement](https://mlsanity.com/paper/2406.07646/) (2024)
- [High-Fidelity Speech Enhancement via Discrete Audio Tokens](https://mlsanity.com/paper/2510.02187/) (2025)
- [Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancement](https://mlsanity.com/paper/2508.20859/) (2025)
- [Scaling Properties of Continuous Diffusion Spoken Language Models](https://mlsanity.com/paper/2604.24416/) (2026)
- [PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction](https://mlsanity.com/paper/2607.08111/) (2026)
- [When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection](https://mlsanity.com/paper/2608.30204/) (2026)
- [Acoustic Echo Cancellation using Residual U-Nets](https://mlsanity.com/paper/2109.09686/) (2021)
- [Speech MOS multi-task learning and rater bias correction](https://mlsanity.com/paper/2212.01911/) (2022)
- [Objective and subjective evaluation of speech enhancement methods in the UDASE task of the 7th CHiME challenge](https://mlsanity.com/paper/2402.01413/) (2024)
- [Crowdsourced Multilingual Speech Intelligibility Testing](https://mlsanity.com/paper/2403.14817/) (2024)
- [A multidimensional measurement of photorealistic avatar quality of experience](https://mlsanity.com/paper/2411.09066/) (2024)
- [Robust Residual Finite Scalar Quantization for Neural Compression](https://mlsanity.com/paper/2508.15860/) (2025)
- [Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditions](https://mlsanity.com/paper/2509.24457/) (2025)
- [PASQA: Pitch-Accent-Focused Speech Quality Assessment Model Trained on Synthetic Speech with Accent Errors](https://mlsanity.com/paper/2606.20137/) (2026)
- [Transcript-Supervised Post-Training of Generative Speech Enhancement on Real Recordings via Reinforce Adjoint Matching](https://mlsanity.com/paper/2609.29405/) (2026)

## Abstract (arXiv)

The ITU-T Recommendation P.808 provides a crowdsourcing approach for conducting a subjective assessment of speech quality using the Absolute Category Rating (ACR) method. We provide an open-source implementation of the ITU-T Rec. P.808 that runs on the Amazon Mechanical Turk platform. We extended our implementation to include Degradation Category Ratings (DCR) and Comparison Category Ratings (CCR) test methods. We also significantly speed up the test process by integrating the participant qualification step into the main rating task compared to a two-stage qualification and rating solution. We provide program scripts for creating and executing the subjective test, and data cleansing and analyzing the answers to avoid operational errors. To validate the implementation, we compare the Mean Opinion Scores (MOS) collected through our implementation with MOS values from a standard laboratory experiment conducted based on the ITU-T Rec. P.800. We also evaluate the reproducibility of the result of the subjective speech quality assessment through crowdsourcing using our implementation. Finally, we quantify the impact of parts of the system designed to improve the reliability: environmental tests, gold and trapping questions, rating patterns, and a headset usage test.