# A scalable noisy speech dataset and online subjective test framework

arXiv: https://arxiv.org/abs/1909.08050 · HTML: https://mlsanity.com/paper/1909.08050/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2019.

## Built on by (most cited first, 30 of 48)

- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [Interspeech 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2101.01902/) (2021)
- [Speech Robust Bench: A Robustness Benchmark For Speech Recognition](https://mlsanity.com/paper/2403.07937/) (2024)
- [RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users](https://mlsanity.com/paper/2504.10445/) (2025)
- [Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting](https://mlsanity.com/paper/2206.15400/) (2022)
- [High Fidelity Speech Regeneration with Application to Speech Enhancement](https://mlsanity.com/paper/2102.00429/) (2021)
- [Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use](https://mlsanity.com/paper/2604.22821/) (2026)
- [Hybrid Neural Networks for On-device Directional Hearing](https://mlsanity.com/paper/2112.05893/) (2021)
- [Accelerating RNN-based Speech Enhancement on a Multi-Core MCU with Mixed FP16-INT8 Post-Training Quantization](https://mlsanity.com/paper/2210.07692/) (2022)
- [InQSS: a speech intelligibility and quality assessment model using a multi-task learning network](https://mlsanity.com/paper/2111.02585/) (2021)
- [Advancing Test-Time Adaptation in Wild Acoustic Test Settings](https://mlsanity.com/paper/2310.09505/) (2023)
- [Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mind](https://mlsanity.com/paper/2409.10849/) (2024)
- [A Systematic Comparison of Phonetic Aware Techniques for Speech Enhancement](https://mlsanity.com/paper/2206.11000/) (2022)
- [Improving Speech Enhancement through Fine-Grained Speech Characteristics](https://mlsanity.com/paper/2207.00237/) (2022)
- [QbyE-MLPMixer: Query-by-Example Open-Vocabulary Keyword Spotting using MLPMixer](https://mlsanity.com/paper/2206.13231/) (2022)
- [Test-Time Training for Speech](https://mlsanity.com/paper/2309.10930/) (2023)
- [Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder](https://mlsanity.com/paper/2309.00140/) (2023)
- [LI-TTA: Language Informed Test-Time Adaptation for Automatic Speech Recognition](https://mlsanity.com/paper/2408.05769/) (2024)
- [Speech Enhancement for Wake-Up-Word detection in Voice Assistants](https://mlsanity.com/paper/2101.12732/) (2021)
- [U-vectors: Generating clusterable speaker embedding from unlabeled data](https://mlsanity.com/paper/2102.03868/) (2021)
- [A Benchmarking on Cloud based Speech-To-Text Services for French Speech and Background Noise Effect](https://mlsanity.com/paper/2105.03409/) (2021)
- [TAPLoss: A Temporal Acoustic Parameter Loss for Speech Enhancement](https://mlsanity.com/paper/2302.08088/) (2023)
- [PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech Enhancement](https://mlsanity.com/paper/2302.08095/) (2023)
- [VAKTA-SETU: A Speech-to-Speech Machine Translation Service in Select Indic Languages](https://mlsanity.com/paper/2305.12518/) (2023)
- [ROSE: A Recognition-Oriented Speech Enhancement Framework in Air Traffic Control Using Multi-Objective Learning](https://mlsanity.com/paper/2312.06118/) (2023)
- [Resource-Efficient Speech Quality Prediction through Quantization Aware Training and Binary Activation Maps](https://mlsanity.com/paper/2407.04578/) (2024)
- [Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition](https://mlsanity.com/paper/2302.09723/) (2023)
- [SECP: A Speech Enhancement-Based Curation Pipeline For Scalable Acquisition Of Clean Speech](https://mlsanity.com/paper/2402.12482/) (2024)
- [Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks](https://mlsanity.com/paper/2404.14024/) (2024)
- [Roadmap towards Superhuman Speech Understanding using Large Language Models](https://mlsanity.com/paper/2410.13268/) (2024)

## Abstract (arXiv)

Background noise is a major source of quality impairments in Voice over Internet Protocol (VoIP) and Public Switched Telephone Network (PSTN) calls. Recent work shows the efficacy of deep learning for noise suppression, but the datasets have been relatively small compared to those used in other domains (e.g., ImageNet) and the associated evaluations have been more focused. In order to better facilitate deep learning research in Speech Enhancement, we present a noisy speech dataset (MS-SNSD) that can scale to arbitrary sizes depending on the number of speakers, noise types, and Speech to Noise Ratio (SNR) levels desired. We show that increasing dataset sizes increases noise suppression performance as expected. In addition, we provide an open-source evaluation methodology to evaluate the results subjectively at scale using crowdsourcing, with a reference algorithm to normalize the results. To demonstrate the dataset and evaluation framework we apply it to several noise suppressors and compare the subjective Mean Opinion Score (MOS) with objective quality measures such as SNR, PESQ, POLQA, and VISQOL and show why MOS is still required. Our subjective MOS evaluation is the first large scale evaluation of Speech Enhancement algorithms that we are aware of.