# SpeechDx: A Multi-Task Benchmark for Clinical Speech AI

arXiv: https://arxiv.org/abs/2606.17339 · HTML: https://mlsanity.com/paper/2606.17339/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 106 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [Alzheimer's Dementia Recognition through Spontaneous Speech: The ADReSS Challenge](https://mlsanity.com/paper/2004.06833/) (2020)
- [Open-Source Conversational AI with SpeechBrain 1.0](https://mlsanity.com/paper/2407.00463/) (2024)
- [A scalable noisy speech dataset and online subjective test framework](https://mlsanity.com/paper/1909.08050/) (2019)
- [Coswara: A respiratory sounds and symptoms dataset for remote screening of SARS-CoV-2 infection](https://mlsanity.com/paper/2305.12741/) (2023)
- [DiCOVA Challenge: Dataset, task, and baseline system for COVID-19 diagnosis using acoustics](https://mlsanity.com/paper/2103.09148/) (2021)
- [KSoF: The Kassel State of Fluency Dataset -- A Therapy Centered Dataset of Stuttering](https://mlsanity.com/paper/2203.05383/) (2022)
- [Multilingual Alzheimer's Dementia Recognition through Spontaneous Speech: a Signal Processing Grand Challenge](https://mlsanity.com/paper/2301.05562/) (2023)
- [Investigating Self-supervised Pretraining Frameworks for Pathological Speech Recognition](https://mlsanity.com/paper/2203.15431/) (2022)
- [OptunaHub: A Platform for Black-Box Optimization](https://mlsanity.com/paper/2510.02798/) (2025)
- [On using the UA-Speech and TORGO databases to validate automatic dysarthric speech classification approaches](https://mlsanity.com/paper/2211.08833/) (2022)
- [WavJEPA: Semantic learning unlocks robust audio foundation models for raw waveforms](https://mlsanity.com/paper/2509.23238/) (2025)
- [The 1st SpeechWellness Challenge: Detecting Suicide Risk Among Adolescents](https://mlsanity.com/paper/2501.06474/) (2025)
- [Classifying Dementia in the Presence of Depression: A Cross-Corpus Study](https://mlsanity.com/paper/2308.08306/) (2023)
- [Probing Whisper for Dysarthric Speech in Detection and Assessment](https://mlsanity.com/paper/2510.04219/) (2025)
- [Supervised and Self-supervised Pretraining Based COVID-19 Detection Using Acoustic Breathing/Cough/Speech Signals](https://mlsanity.com/paper/2201.08934/) (2022)

## Abstract (arXiv)

Speech offers a uniquely informative window into health by simultaneously engaging neurological, motor, respiratory, and vocal systems. Current clinical speech AI methods have largely progressed through isolated condition-specific studies, making results difficult to compare and generalization difficult to assess. We introduce SpeechDx, a large-scale benchmark for clinical speech AI spanning 12 datasets and 27 tasks across diverse health conditions. To enable evaluation across shared clinical mechanisms, SpeechDx structures tasks by the stage of speech production they disrupt: conceptualization, formulation, and articulation. The benchmark tests generalization by including tasks with limited labeled data and evaluating the same health condition across multiple datasets, distinguishing clinically meaningful patterns from dataset artefacts. We systematically evaluate 12 state-of-the-art audio encoders across all tasks and under zero-shot cross-condition transfer. Results show that large-scale speech models represent the strongest overall baselines, domain-specific models improve performance only on closely matched tasks, and no current representation generalizes reliably across the clinical speech landscape. SpeechDx establishes a shared evaluation framework for tracking progress toward general-purpose clinical speech representations