# QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions

arXiv: https://arxiv.org/abs/2503.20290 · HTML: https://mlsanity.com/paper/2503.20290/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 53 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 13 of 13)

- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [AudioBench: A Universal Benchmark for Audio Large Language Models](https://mlsanity.com/paper/2406.16020/) (2024)
- [Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks](https://mlsanity.com/paper/2411.05361/) (2024)
- [OpenVoice: Versatile Instant Voice Cloning](https://mlsanity.com/paper/2312.01479/) (2023)
- [Audio Entailment: Assessing Deductive Reasoning for Audio Understanding](https://mlsanity.com/paper/2407.18062/) (2024)
- [Can Large Language Models Understand Spatial Audio?](https://mlsanity.com/paper/2406.07914/) (2024)
- [Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation](https://mlsanity.com/paper/2409.16644/) (2024)
- [DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understanding](https://mlsanity.com/paper/2406.09345/) (2024)
- [A Study on Zero-shot Non-intrusive Speech Assessment using Large Language Models](https://mlsanity.com/paper/2409.09914/) (2024)

## Built on by (most cited first, 13 of 13)

- [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025)
- [SpeechJudge: Towards Human-Level Judgment for Speech Naturalness](https://mlsanity.com/paper/2511.07931/) (2025)
- [Interpretable All-Type Audio Deepfake Detection with Audio LLMs via Frequency-Time Reinforcement Learning](https://mlsanity.com/paper/2601.02983/) (2026)
- [AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following](https://mlsanity.com/paper/2606.03116/) (2026)
- [JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions](https://mlsanity.com/paper/2605.04505/) (2026)
- [MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models](https://mlsanity.com/paper/2608.22236/) (2026)
- [Calibration-Reasoning Framework for Descriptive Speech Quality Assessment](https://mlsanity.com/paper/2603.10175/) (2026)
- [SpeakerSleuth: Can Large Audio-Language Models Judge Speaker Consistency across Multi-turn Dialogues?](https://mlsanity.com/paper/2601.04029/) (2026)
- [LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancement](https://mlsanity.com/paper/2603.13952/) (2026)
- [Augmenting Dysarthric Speech Severity Assessment with MOS Supervision](https://mlsanity.com/paper/2606.18645/) (2026)
- [Assessing Quality of Experience in Natural Language Generation of German Text](https://mlsanity.com/paper/2608.18888/) (2026)
- [AcoustiClaim: A Numeric Claim Benchmark with Instrument Ground Truth](https://mlsanity.com/paper/2609.30483/) (2026)
- [SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferences](https://mlsanity.com/paper/2609.34582/) (2026)

## Abstract (arXiv)

This paper explores a novel perspective to speech quality assessment by leveraging natural language descriptions, offering richer, more nuanced insights than traditional numerical scoring methods. Natural language feedback provides instructive recommendations and detailed evaluations, yet existing datasets lack the comprehensive annotations needed for this approach. To bridge this gap, we introduce QualiSpeech, a comprehensive low-level speech quality assessment dataset encompassing 11 key aspects and detailed natural language comments that include reasoning and contextual insights. Additionally, we propose the QualiSpeech Benchmark to evaluate the low-level speech understanding capabilities of auditory large language models (LLMs). Experimental results demonstrate that finetuned auditory LLMs can reliably generate detailed descriptions of noise and distortion, effectively identifying their types and temporal characteristics. The results further highlight the potential for incorporating reasoning to enhance the accuracy and reliability of quality assessments. The dataset will be released at https://huggingface.co/datasets/tsinghua-ee/QualiSpeech.