# What does BERT learn about prosody?

arXiv: https://arxiv.org/abs/2304.12706 · HTML: https://mlsanity.com/paper/2304.12706/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://mlsanity.com/paper/1808.08949/) (2018)
- [Predicting Prosodic Prominence from Text with Pre-trained Contextualized Word Representations](https://mlsanity.com/paper/1908.02262/) (2019)
- [Towards Interpreting BERT for Reading Comprehension Based QA](https://mlsanity.com/paper/2010.08983/) (2020)
- [BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model](https://mlsanity.com/paper/2207.01718/) (2022)
- [Using Integrated Gradients and Constituency Parse Trees to explain Linguistic Acceptability learnt by BERT](https://mlsanity.com/paper/2106.07349/) (2021)

## Built on by (most cited first, 4 of 4)

- [Investigating the Utility of Surprisal from Large Language Models for Speech Synthesis Prosody](https://mlsanity.com/paper/2306.09814/) (2023)
- [Representation of perceived prosodic similarity of conversational feedback](https://mlsanity.com/paper/2505.13268/) (2025)
- [Word-wise intonation model for cross-language TTS systems](https://mlsanity.com/paper/2409.20374/) (2024)
- [Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility](https://mlsanity.com/paper/2505.16277/) (2025)

## Abstract (arXiv)

Language models have become nearly ubiquitous in natural language processing applications achieving state-of-the-art results in many tasks including prosody. As the model design does not define predetermined linguistic targets during training but rather aims at learning generalized representations of the language, analyzing and interpreting the representations that models implicitly capture is important in bridging the gap between interpretability and model performance. Several studies have explored the linguistic information that models capture providing some insights on their representational capacity. However, the current studies have not explored whether prosody is part of the structural information of the language that models learn. In this work, we perform a series of experiments on BERT probing the representations captured at different layers. Our results show that information about prosodic prominence spans across many layers but is mostly focused in middle layers suggesting that BERT relies mostly on syntactic and semantic information.