mlsanityData to 7 Oct 2026

Voice Activity Projection: Self-supervised Learning of Turn-taking Events

arXiv 2205.09812 ↗First posted 19 May 2022eess.AS cs.SDcited by 54

Not on the map. Why?

Abstract (arXiv)

The modeling of turn-taking in dialog can be viewed as the modeling of the dynamics of voice activity of the interlocutors. We extend prior work and define the predictive task of Voice Activity Projection, a general, self-supervised objective, as a way to train turn-taking models without the need of labeled data. We highlight a theoretical weakness with prior approaches, arguing for the need of modeling the dependency of voice activity events in the projection window. We propose four zero-shot tasks, related to the prediction of upcoming turn-shifts and backchannels, and show that the proposed model outperforms prior work.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 May 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 14; 2026: 30.

54 papers.

Sort:

By area:

  1. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
  2. 2024Multilingual Turn-taking Prediction Using Voice Activity Projectioncited by 26
  3. 2024Real-time and Continuous Turn-taking Prediction Using Voice Activity Projectioncited by 21
  4. 2024Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusioncited by 20
  5. 2025Applying General Turn-taking Models to Conversational Human-Robot Interactioncited by 13
  6. 2024Large Language Models Know What To Say But Not When To Speakcited by 11
  7. 2025Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signalscited by 10
  8. 2026Multi-Faceted Interactivity Alignment in Full-Duplex Speech Modelscited by 9
  9. 2024Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projectioncited by 8
  10. 2025Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interactioncited by 8
  11. 2025EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wildcited by 7
  12. 2025A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experimentcited by 6
38 more
  1. 2025Speculative End-Turn Detector for Efficient Speech Chatbot Assistantcited by 5
  2. 2024Target conversation extraction: Source separation using turn-taking dynamicscited by 4
  3. 2025Triadic Multi-party Voice Activity Projection for Turn-taking in Spoken Dialogue Systemscited by 4
  4. 2026TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialoguecited by 4
  5. 2023Modeling Speaker-Listener Interaction for Backchannel Predictioncited by 3
  6. 2023What makes a good pause? Investigating the turn-holding effects of fillerscited by 3
  7. 2025Visual Cues Support Robust Turn-taking Prediction in Noisecited by 3
  8. 2025"Dyadosyncrasy", Idiosyncrasy and Demographic Factors in Turn-Takingcited by 2
  9. 2025Voice Activity Projection Model with Multimodal Encoderscited by 2
  10. 2026DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretrainingcited by 2
  11. 2025Prompt-Guided Turn-Taking Predictioncited by 1
  12. 2025Proactive Hearing Assistants that Isolate Egocentric Conversationscited by 1
  13. 2025Multilingual and Continuous Backchannel Prediction: A Cross-lingual Studycited by 1
  14. 2026The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representationscited by 1
  15. 2026JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systemscited by 1
  16. 2026MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wildcited by 1
  17. 2026TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialoguecited by 1
  18. 2026Full-Duplex Speech Models Take the Floor When Asked, Not When Neededcited by 1
  19. 2024Learning to Generate Context-Sensitive Backchannel Smiles for Embodied AI Agents with Applications in Mental Health Dialoguescited by 0
  20. 2024NaturalTurn: A Method to Segment Speech into Psychologically Meaningful Conversational Turnscited by 0
  21. 2026S-MARC: Causal Streaming Reasoning for Full-Duplex Conversational Behavior Modelingcited by 0
  22. 2026A Hierarchical End-of-Turn Model with Primary Speaker Segmentation for Real-Time Conversational AIcited by 0
  23. 2026Toward Signing Activity Projection in Sign Language Interactioncited by 0
  24. 2026Fast When, Careful Who: Dual-Process Multiparty Turn-Taking with Diffusion Augmentationcited by 0
  25. 2026Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetingscited by 0
  26. 2026Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoderscited by 0
  27. 2026Low-Latency Turn-Taking via Context-Aware Preface Generation in a Real-World Dialogue Robotcited by 0
  28. 2026CallScreenBench: Benchmarking Small Language Models as Phone Secretariescited by 0
  29. 2026Real-TurnTurk: A Multimodal Turkish Corpus for Turn-Taking Predictioncited by 0
  30. 2026Conversation Coach: A Voice-enabled AI System that Helps Practice Difficult Workplace Conversationscited by 0
  31. 2026Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialoguecited by 0
  32. 2026TamilEOT: A Dataset and Model for Semantic End-of-Turn Detection in Tamil Telephone Speechcited by 0
  33. 2026Using Semantic Uncertainty to Estimate Transition Relevance in Turn-takingcited by 0
  34. 2026Less can be More: What Aspects of Speech Drive End-of-Turn Detectioncited by 0
  35. 2026Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projectioncited by 0
  36. 2026Audio-Visual Turn-taking Prediction in Cocktail Party Scenarioscited by 0
  37. 2026Voice-Light: A Full-Duplex Cascaded Voice Agent with Causal Turn-Taking and Speculative Generationcited by 0
  38. 2026Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Modelscited by 0

50 most cited of 54 All 54 (JSON)