# The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations

arXiv: https://arxiv.org/abs/2601.13835 · HTML: https://mlsanity.com/paper/2601.13835/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Builds on (most cited first, 13 of 13)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Voice Activity Projection: Self-supervised Learning of Turn-taking Events](https://mlsanity.com/paper/2205.09812/) (2022)
- [Intelligible Lip-to-Speech Synthesis with Speech Units](https://mlsanity.com/paper/2305.19603/) (2023)
- [Multilingual Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2403.06487/) (2024)
- [How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models](https://mlsanity.com/paper/2209.05161/) (2022)
- [Investigating Speech Features for Continuous Turn-Taking Prediction Using LSTMs](https://mlsanity.com/paper/1806.11461/) (2018)
- [When can I Speak? Predicting initiation points for spoken dialogue agents](https://mlsanity.com/paper/2208.03812/) (2022)
- [Unveiling Interpretability in Self-Supervised Speech Representations for Parkinson's Diagnosis](https://mlsanity.com/paper/2412.02006/) (2024)
- [Visual Cues Support Robust Turn-taking Prediction in Noise](https://mlsanity.com/paper/2505.22088/) (2025)
- [Sounding Like a Winner? Prosodic Differences in Post-Match Interviews](https://mlsanity.com/paper/2506.02283/) (2025)

## Built on by (most cited first, 1 of 1)

- [Less can be More: What Aspects of Speech Drive End-of-Turn Detection](https://mlsanity.com/paper/2609.11066/) (2026)

## Abstract (arXiv)

Fluid turn-taking remains a key challenge in human-robot interaction. Self-supervised speech representations (S3Rs) have driven many advances, but it remains unclear whether S3R-based turn-taking models rely on prosodic cues, lexical cues or both. We introduce a vocoder-based approach to control prosody and lexical cues in speech more cleanly than prior work. This allows us to probe the voice-activity projection model, an S3R-based turn-taking model. We find that prediction on prosody-matched, unintelligible noise is similar to accuracy on clean speech. This reveals both prosodic and lexical cues support turn-taking, but either can be used in isolation. Hence, future models may only require prosody, providing privacy and potential performance benefits. When either prosodic or lexical information is disrupted, the model exploits the other without further training, indicating they are encoded in S3Rs with limited interdependence. Results are consistent in CPC-based and wav2vec2.0 S3Rs. We discuss our findings and highlight a number of directions for future work. All code is available to support future research.