# Multilingual Turn-taking Prediction Using Voice Activity Projection

arXiv: https://arxiv.org/abs/2403.06487 · HTML: https://mlsanity.com/paper/2403.06487/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 43 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 7 of 7)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Voice Activity Projection: Self-supervised Learning of Turn-taking Events](https://mlsanity.com/paper/2205.09812/) (2022)
- [TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog](https://mlsanity.com/paper/2010.10874/) (2020)
- [How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models](https://mlsanity.com/paper/2209.05161/) (2022)
- [Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2401.04868/) (2024)

## Built on by (most cited first, 26 of 26)

- [Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics](https://mlsanity.com/paper/2503.01174/) (2025)
- [Large Language Models Know What To Say But Not When To Speak](https://mlsanity.com/paper/2410.16044/) (2024)
- [Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection](https://mlsanity.com/paper/2410.15929/) (2024)
- [Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction](https://mlsanity.com/paper/2505.21043/) (2025)
- [TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue](https://mlsanity.com/paper/2608.25218/) (2026)
- [A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment](https://mlsanity.com/paper/2503.06241/) (2025)
- [Triadic Multi-party Voice Activity Projection for Turn-taking in Spoken Dialogue Systems](https://mlsanity.com/paper/2507.07518/) (2025)
- [Human-Like Embodied AI Interviewer: Employing Android ERICA in Real International Conference](https://mlsanity.com/paper/2412.09867/) (2024)
- [Visual Cues Support Robust Turn-taking Prediction in Noise](https://mlsanity.com/paper/2505.22088/) (2025)
- ["Dyadosyncrasy", Idiosyncrasy and Demographic Factors in Turn-Taking](https://mlsanity.com/paper/2505.24736/) (2025)
- [Voice Activity Projection Model with Multimodal Encoders](https://mlsanity.com/paper/2506.03980/) (2025)
- [Prompt-Guided Turn-Taking Prediction](https://mlsanity.com/paper/2506.21191/) (2025)
- [Multilingual and Continuous Backchannel Prediction: A Cross-lingual Study](https://mlsanity.com/paper/2512.14085/) (2025)
- [The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations](https://mlsanity.com/paper/2601.13835/) (2026)
- [JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems](https://mlsanity.com/paper/2603.26515/) (2026)
- [TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue](https://mlsanity.com/paper/2607.01345/) (2026)
- [MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation](https://mlsanity.com/paper/2609.36903/) (2026)
- [Everything Counts: The Managed Omnirelevance of Speech in Human-Voice Agent Interaction](https://mlsanity.com/paper/2510.22610/) (2025)
- [A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AI](https://mlsanity.com/paper/2603.13379/) (2026)
- [Toward Signing Activity Projection in Sign Language Interaction](https://mlsanity.com/paper/2606.09424/) (2026)
- [Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders](https://mlsanity.com/paper/2607.07294/) (2026)
- [Does Listening Matter? Backchanneling and Nodding in AI Clone](https://mlsanity.com/paper/2608.19527/) (2026)
- [Using Semantic Uncertainty to Estimate Transition Relevance in Turn-taking](https://mlsanity.com/paper/2609.10934/) (2026)
- [Audio-Visual Turn-taking Prediction in Cocktail Party Scenarios](https://mlsanity.com/paper/2609.17056/) (2026)
- [Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Models](https://mlsanity.com/paper/2609.27372/) (2026)
- [Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes](https://mlsanity.com/paper/2609.30798/) (2026)

## Abstract (arXiv)

This paper investigates the application of voice activity projection (VAP), a predictive turn-taking model for spoken dialogue, on multilingual data, encompassing English, Mandarin, and Japanese. The VAP model continuously predicts the upcoming voice activities of participants in dyadic dialogue, leveraging a cross-attention Transformer to capture the dynamic interplay between participants. The results show that a monolingual VAP model trained on one language does not make good predictions when applied to other languages. However, a multilingual model, trained on all three languages, demonstrates predictive performance on par with monolingual models across all languages. Further analyses show that the multilingual model has learned to discern the language of the input signal. We also analyze the sensitivity to pitch, a prosodic cue that is thought to be important for turn-taking. Finally, we compare two different audio encoders, contrastive predictive coding (CPC) pre-trained on English, with a recent model based on multilingual wav2vec 2.0 (MMS).