# ProsAudit, a prosodic benchmark for self-supervised speech models

arXiv: https://arxiv.org/abs/2302.12057 · HTML: https://mlsanity.com/paper/2302.12057/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 21 references that are in the map.
- First posted on arXiv 23 Feb 2023.

## Builds on (most cited first, 12 of 12)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [The Zero Resource Speech Challenge 2021: Spoken language modelling](https://mlsanity.com/paper/2104.14700/) (2021)
- [The Zero Resource Speech Challenge 2017](https://mlsanity.com/paper/1712.04313/) (2017)
- [On the Utility of Self-supervised Models for Prosody-related Tasks](https://mlsanity.com/paper/2210.07185/) (2022)

## Built on by (most cited first, 25 of 25)

- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Salmon: A Suite for Acoustic Language Model Evaluation](https://mlsanity.com/paper/2409.07437/) (2024)
- [EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models](https://mlsanity.com/paper/2312.14069/) (2023)
- [Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?](https://mlsanity.com/paper/2410.24019/) (2024)
- [SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision](https://mlsanity.com/paper/2512.20308/) (2025)
- [Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models](https://mlsanity.com/paper/2505.17446/) (2025)
- [Prosodic Structure Beyond Lexical Content: A Study of Self-Supervised Learning](https://mlsanity.com/paper/2506.02584/) (2025)
- [Representation of perceived prosodic similarity of conversational feedback](https://mlsanity.com/paper/2505.13268/) (2025)
- [Pitch Accent Detection improves Pretrained Automatic Speech Recognition](https://mlsanity.com/paper/2508.04814/) (2025)
- [Benchmarking Prosody Encoding in Discrete Speech Tokens](https://mlsanity.com/paper/2508.11224/) (2025)
- [Knowing What to Stress: A Discourse-Conditioned Text-to-Speech Benchmark](https://mlsanity.com/paper/2604.10580/) (2026)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)
- [Linguists should learn to love speech-based deep learning models](https://mlsanity.com/paper/2512.14506/) (2025)
- [Are Language Models Models?](https://mlsanity.com/paper/2601.10421/) (2026)
- [Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilities](https://mlsanity.com/paper/2604.09466/) (2026)
- [You Can't Fight in Here! This is BBS!](https://mlsanity.com/paper/2604.09501/) (2026)
- [Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear](https://mlsanity.com/paper/2605.10061/) (2026)
- [Do Speech Emphasis Models Generalize across Languages and Emotions?](https://mlsanity.com/paper/2606.27717/) (2026)
- [Prosody-driven Jailbreaks in Audio LLMs: A Controlled Study and Mechanistic Analysis](https://mlsanity.com/paper/2607.26541/) (2026)
- [Language Discrimination Improves Linguistic Learning in Multilingual Speech Models](https://mlsanity.com/paper/2609.33345/) (2026)
- [Does a prosody-trained representation help beyond trainable fusion? A parameter-matched study with frozen HuBERT](https://mlsanity.com/paper/2609.36754/) (2026)
- [Can Prosodic Style Be Inferred from Text Alone? Evidence from Unsupervised Acoustic Clusters](https://mlsanity.com/paper/2610.05575/) (2026)

## Abstract (arXiv)

We present ProsAudit, a benchmark in English to assess structural prosodic knowledge in self-supervised learning (SSL) speech models. It consists of two subtasks, their corresponding metrics, and an evaluation dataset. In the protosyntax task, the model must correctly identify strong versus weak prosodic boundaries. In the lexical task, the model needs to correctly distinguish between pauses inserted between words and within words. We also provide human evaluation scores on this benchmark. We evaluated a series of SSL models and found that they were all able to perform above chance on both tasks, even when evaluated on an unseen language. However, non-native models performed significantly worse than native ones on the lexical task, highlighting the importance of lexical knowledge in this task. We also found a clear effect of size with models trained on more data performing better in the two subtasks.