# Voice Activity Projection: Self-supervised Learning of Turn-taking Events

arXiv: https://arxiv.org/abs/2205.09812 · HTML: https://mlsanity.com/paper/2205.09812/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2022.

## Built on by (most cited first, 30 of 60)

- [Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics](https://mlsanity.com/paper/2503.01174/) (2025)
- [Multilingual Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2403.06487/) (2024)
- [Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2401.04868/) (2024)
- [Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion](https://mlsanity.com/paper/2401.14717/) (2024)
- [Applying General Turn-taking Models to Conversational Human-Robot Interaction](https://mlsanity.com/paper/2501.08946/) (2025)
- [Large Language Models Know What To Say But Not When To Speak](https://mlsanity.com/paper/2410.16044/) (2024)
- [Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals](https://mlsanity.com/paper/2505.12654/) (2025)
- [Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models](https://mlsanity.com/paper/2606.11167/) (2026)
- [Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection](https://mlsanity.com/paper/2410.15929/) (2024)
- [Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction](https://mlsanity.com/paper/2505.21043/) (2025)
- [EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild](https://mlsanity.com/paper/2502.14892/) (2025)
- [TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue](https://mlsanity.com/paper/2608.25218/) (2026)
- [A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment](https://mlsanity.com/paper/2503.06241/) (2025)
- [Speculative End-Turn Detector for Efficient Speech Chatbot Assistant](https://mlsanity.com/paper/2503.23439/) (2025)
- [Target conversation extraction: Source separation using turn-taking dynamics](https://mlsanity.com/paper/2407.11277/) (2024)
- [Triadic Multi-party Voice Activity Projection for Turn-taking in Spoken Dialogue Systems](https://mlsanity.com/paper/2507.07518/) (2025)
- [DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining](https://mlsanity.com/paper/2603.08216/) (2026)
- [Modeling Speaker-Listener Interaction for Backchannel Prediction](https://mlsanity.com/paper/2304.04472/) (2023)
- [What makes a good pause? Investigating the turn-holding effects of fillers](https://mlsanity.com/paper/2305.02101/) (2023)
- [Visual Cues Support Robust Turn-taking Prediction in Noise](https://mlsanity.com/paper/2505.22088/) (2025)
- ["Dyadosyncrasy", Idiosyncrasy and Demographic Factors in Turn-Taking](https://mlsanity.com/paper/2505.24736/) (2025)
- [Voice Activity Projection Model with Multimodal Encoders](https://mlsanity.com/paper/2506.03980/) (2025)
- [MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild](https://mlsanity.com/paper/2606.16731/) (2026)
- [NaturalTurn: A Method to Segment Speech into Psychologically Meaningful Conversational Turns](https://mlsanity.com/paper/2403.15615/) (2024)
- [Prompt-Guided Turn-Taking Prediction](https://mlsanity.com/paper/2506.21191/) (2025)
- [Proactive Hearing Assistants that Isolate Egocentric Conversations](https://mlsanity.com/paper/2511.11473/) (2025)
- [Multilingual and Continuous Backchannel Prediction: A Cross-lingual Study](https://mlsanity.com/paper/2512.14085/) (2025)
- [The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations](https://mlsanity.com/paper/2601.13835/) (2026)
- [JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems](https://mlsanity.com/paper/2603.26515/) (2026)
- [TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue](https://mlsanity.com/paper/2607.01345/) (2026)

## Abstract (arXiv)

The modeling of turn-taking in dialog can be viewed as the modeling of the dynamics of voice activity of the interlocutors. We extend prior work and define the predictive task of Voice Activity Projection, a general, self-supervised objective, as a way to train turn-taking models without the need of labeled data. We highlight a theoretical weakness with prior approaches, arguing for the need of modeling the dependency of voice activity events in the projection window. We propose four zero-shot tasks, related to the prediction of upcoming turn-shifts and backchannels, and show that the proposed model outperforms prior work.