# How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models

arXiv: https://arxiv.org/abs/2209.05161 · HTML: https://mlsanity.com/paper/2209.05161/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2022.

## Built on by (most cited first, 23 of 23)

- [Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics](https://mlsanity.com/paper/2503.01174/) (2025)
- [Multilingual Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2403.06487/) (2024)
- [Applying General Turn-taking Models to Conversational Human-Robot Interaction](https://mlsanity.com/paper/2501.08946/) (2025)
- [Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals](https://mlsanity.com/paper/2505.12654/) (2025)
- [Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection](https://mlsanity.com/paper/2410.15929/) (2024)
- [Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction](https://mlsanity.com/paper/2505.21043/) (2025)
- [The Role of Prosody in Spoken Question Answering](https://mlsanity.com/paper/2502.05389/) (2025)
- [A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment](https://mlsanity.com/paper/2503.06241/) (2025)
- [Speak or Stay Silent: Context-Aware Turn-Taking in Multi-Party Dialogue](https://mlsanity.com/paper/2603.11409/) (2026)
- [Prosodic Structure Beyond Lexical Content: A Study of Self-Supervised Learning](https://mlsanity.com/paper/2506.02584/) (2025)
- [Response-conditioned Turn-taking Prediction](https://mlsanity.com/paper/2305.02036/) (2023)
- [What makes a good pause? Investigating the turn-holding effects of fillers](https://mlsanity.com/paper/2305.02101/) (2023)
- [Visual Cues Support Robust Turn-taking Prediction in Noise](https://mlsanity.com/paper/2505.22088/) (2025)
- [3M-TRANSFORMER: A Multi-Stage Multi-Stream Multimodal Transformer for Embodied Turn-Taking Prediction](https://mlsanity.com/paper/2310.14859/) (2023)
- [Voice Activity Projection Model with Multimodal Encoders](https://mlsanity.com/paper/2506.03980/) (2025)
- [MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild](https://mlsanity.com/paper/2606.16731/) (2026)
- [Modeling Turn-Taking with Semantically Informed Gestures](https://mlsanity.com/paper/2510.19350/) (2025)
- [The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations](https://mlsanity.com/paper/2601.13835/) (2026)
- [Everything Counts: The Managed Omnirelevance of Speech in Human-Voice Agent Interaction](https://mlsanity.com/paper/2510.22610/) (2025)
- [Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection](https://mlsanity.com/paper/2609.14666/) (2026)
- [Audio-Visual Turn-taking Prediction in Cocktail Party Scenarios](https://mlsanity.com/paper/2609.17056/) (2026)
- [Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes](https://mlsanity.com/paper/2609.30798/) (2026)
- [Multi-Party Backchannel Prediction: a Diagnosis, a Benchmark, and a Ceiling](https://mlsanity.com/paper/2610.01488/) (2026)

## Abstract (arXiv)

Turn-taking is a fundamental aspect of human communication and can be described as the ability to take turns, project upcoming turn shifts, and supply backchannels at appropriate locations throughout a conversation. In this work, we investigate the role of prosody in turn-taking using the recently proposed Voice Activity Projection model, which incrementally models the upcoming speech activity of the interlocutors in a self-supervised manner, without relying on explicit annotation of turn-taking events, or the explicit modeling of prosodic features. Through manipulation of the speech signal, we investigate how these models implicitly utilize prosodic information. We show that these systems learn to utilize various prosodic aspects of speech both on aggregate quantitative metrics of long-form conversations and on single utterances specifically designed to depend on prosody.