# Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection

arXiv: https://arxiv.org/abs/2401.04868 · HTML: https://mlsanity.com/paper/2401.04868/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 20 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Builds on (most cited first, 3 of 3)

- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Voice Activity Projection: Self-supervised Learning of Turn-taking Events](https://mlsanity.com/paper/2205.09812/) (2022)
- [TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog](https://mlsanity.com/paper/2010.10874/) (2020)

## Built on by (most cited first, 21 of 21)

- [Multilingual Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2403.06487/) (2024)
- [Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection](https://mlsanity.com/paper/2410.15929/) (2024)
- [TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue](https://mlsanity.com/paper/2608.25218/) (2026)
- [A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment](https://mlsanity.com/paper/2503.06241/) (2025)
- [Target conversation extraction: Source separation using turn-taking dynamics](https://mlsanity.com/paper/2407.11277/) (2024)
- [Human-Like Embodied AI Interviewer: Employing Android ERICA in Real International Conference](https://mlsanity.com/paper/2412.09867/) (2024)
- [Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Models](https://mlsanity.com/paper/2501.00953/) (2025)
- [Voice Activity Projection Model with Multimodal Encoders](https://mlsanity.com/paper/2506.03980/) (2025)
- [Investigating the Effects of Large-Scale Pseudo-Stereo Data and Different Speech Foundation Model on Dialogue Generative Spoken Language Model](https://mlsanity.com/paper/2407.01911/) (2024)
- [Proactive Hearing Assistants that Isolate Egocentric Conversations](https://mlsanity.com/paper/2511.11473/) (2025)
- [JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems](https://mlsanity.com/paper/2603.26515/) (2026)
- [TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue](https://mlsanity.com/paper/2607.01345/) (2026)
- [Everything Counts: The Managed Omnirelevance of Speech in Human-Voice Agent Interaction](https://mlsanity.com/paper/2510.22610/) (2025)
- [Toward Signing Activity Projection in Sign Language Interaction](https://mlsanity.com/paper/2606.09424/) (2026)
- [Fast When, Careful Who: Dual-Process Multiparty Turn-Taking with Diffusion Augmentation](https://mlsanity.com/paper/2606.16568/) (2026)
- [Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders](https://mlsanity.com/paper/2607.07294/) (2026)
- [Low-Latency Turn-Taking via Context-Aware Preface Generation in a Real-World Dialogue Robot](https://mlsanity.com/paper/2607.23204/) (2026)
- [Real-TurnTurk: A Multimodal Turkish Corpus for Turn-Taking Prediction](https://mlsanity.com/paper/2608.22071/) (2026)
- [Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection](https://mlsanity.com/paper/2609.14666/) (2026)
- [Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Models](https://mlsanity.com/paper/2609.27372/) (2026)
- [Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes](https://mlsanity.com/paper/2609.30798/) (2026)

## Abstract (arXiv)

A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialogue stereo audio to future voice activities. The VAP model includes contrastive predictive coding (CPC) and self-attention transformers, followed by a cross-attention transformer. We examine the effect of the input context audio length and demonstrate that the proposed system can operate in real-time with CPU settings, with minimal performance degradation.