# Prosodic Structure Beyond Lexical Content: A Study of Self-Supervised Learning

arXiv: https://arxiv.org/abs/2506.02584 · HTML: https://mlsanity.com/paper/2506.02584/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 53 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [PMI-Masking: Principled masking of correlated spans](https://mlsanity.com/paper/2010.01825/) (2020)
- [ProsAudit, a prosodic benchmark for self-supervised speech models](https://mlsanity.com/paper/2302.12057/) (2023)
- [How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models](https://mlsanity.com/paper/2209.05161/) (2022)
- [Prosodic Prominence and Boundaries in Sequence-to-Sequence Speech Synthesis](https://mlsanity.com/paper/2006.15967/) (2020)
- [Automatic recognition of suprasegmentals in speech](https://mlsanity.com/paper/2108.01122/) (2021)
- [Learning De-identified Representations of Prosody from Raw Audio](https://mlsanity.com/paper/2107.08248/) (2021)
- [Disentangling Prosody Representations with Unsupervised Speech Reconstruction](https://mlsanity.com/paper/2212.06972/) (2022)
- [Do Prosody Transfer Models Transfer Prosody?](https://mlsanity.com/paper/2303.04289/) (2023)
- [Quantifying the perceptual value of lexical and non-lexical channels in speech](https://mlsanity.com/paper/2307.03534/) (2023)

## Built on by (most cited first, 4 of 4)

- [TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems](https://mlsanity.com/paper/2506.19441/) (2025)
- [What Do Prosody and Text Convey? Characterizing How Meaningful Information is Distributed Across Multiple Channels](https://mlsanity.com/paper/2512.16832/) (2025)
- [Lexical Tone is Hard to Quantize: Probing Discrete Speech Units in Mandarin and Yor`ub'a](https://mlsanity.com/paper/2604.07467/) (2026)
- [DiffAnon: Diffusion-based Prosody Control for Voice Anonymization](https://mlsanity.com/paper/2604.26281/) (2026)

## Abstract (arXiv)

People exploit the predictability of lexical structures during text comprehension. Though predictable structure is also present in speech, the degree to which prosody, e.g. intonation, tempo, and loudness, contributes to such structure independently of the lexical content is unclear. This study leverages self-supervised learning (SSL) to examine the temporal granularity of structures in the acoustic correlates of prosody. Representations from our proposed Masked Prosody Model can predict perceptual labels dependent on local information, such as word boundaries, but provide the most value for labels involving longer-term structures, like emotion recognition. Probing experiments across various perceptual labels show strong relative gains over untransformed pitch, energy, and voice activity features. Our results reveal the importance of SSL training objective timescale and highlight the value of complex SSL-encoded structures compared to more constrained classical structures.