# Applying General Turn-taking Models to Conversational Human-Robot Interaction

arXiv: https://arxiv.org/abs/2501.08946 · HTML: https://mlsanity.com/paper/2501.08946/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 60 references that are in the map.
- First posted on arXiv 15 Jan 2025.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- [Voice Activity Projection: Self-supervised Learning of Turn-taking Events](https://mlsanity.com/paper/2205.09812/) (2022)
- [TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog](https://mlsanity.com/paper/2010.10874/) (2020)
- [Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems](https://mlsanity.com/paper/2205.15060/) (2022)
- [How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models](https://mlsanity.com/paper/2209.05161/) (2022)
- [Yeah, Right, Uh-Huh: A Deep Learning Backchannel Predictor](https://mlsanity.com/paper/1706.01340/) (2017)
- [Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue](https://mlsanity.com/paper/2204.10172/) (2022)
- [Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis](https://mlsanity.com/paper/2305.17971/) (2023)
- [A Survey on Dialogue Management in Human-Robot Interaction](https://mlsanity.com/paper/2307.10897/) (2023)
- [What makes a good pause? Investigating the turn-holding effects of fillers](https://mlsanity.com/paper/2305.02101/) (2023)

## Built on by (most cited first, 13 of 13)

- [A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experiment](https://mlsanity.com/paper/2503.06241/) (2025)
- [Thai Semantic End-of-Turn Detection for Real-Time Voice Agents](https://mlsanity.com/paper/2510.04016/) (2025)
- [MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild](https://mlsanity.com/paper/2606.16731/) (2026)
- [Comparing Self-Disclosure Themes and Semantics to a Human, a Robot, and a Disembodied Agent](https://mlsanity.com/paper/2504.06374/) (2025)
- [Human-Human-AI Triadic Programming: Uncovering the Role of AI Agent and the Value of Human Partner in Collaborative Learning](https://mlsanity.com/paper/2601.12134/) (2026)
- [JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems](https://mlsanity.com/paper/2603.26515/) (2026)
- [Everything Counts: The Managed Omnirelevance of Speech in Human-Voice Agent Interaction](https://mlsanity.com/paper/2510.22610/) (2025)
- [Toward Signing Activity Projection in Sign Language Interaction](https://mlsanity.com/paper/2606.09424/) (2026)
- [Responsible Personalisation: The Double-Edged Sword of Personalisation in Human-Robot Interaction](https://mlsanity.com/paper/2607.06344/) (2026)
- [Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders](https://mlsanity.com/paper/2607.07294/) (2026)
- [Low-Latency Turn-Taking via Context-Aware Preface Generation in a Real-World Dialogue Robot](https://mlsanity.com/paper/2607.23204/) (2026)
- [Why did My Robot Just Change Personality? Prompting Guidelines for a Grounded Robot Persona in LLM-Based HRI](https://mlsanity.com/paper/2608.26182/) (2026)
- [Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection](https://mlsanity.com/paper/2609.14666/) (2026)

## Abstract (arXiv)

Turn-taking is a fundamental aspect of conversation, but current Human-Robot Interaction (HRI) systems often rely on simplistic, silence-based models, leading to unnatural pauses and interruptions. This paper investigates, for the first time, the application of general turn-taking models, specifically TurnGPT and Voice Activity Projection (VAP), to improve conversational dynamics in HRI. These models are trained on human-human dialogue data using self-supervised learning objectives, without requiring domain-specific fine-tuning. We propose methods for using these models in tandem to predict when a robot should begin preparing responses, take turns, and handle potential interruptions. We evaluated the proposed system in a within-subject study against a traditional baseline system, using the Furhat robot with 39 adults in a conversational setting, in combination with a large language model for autonomous response generation. The results show that participants significantly prefer the proposed system, and it significantly reduces response delays and interruptions.