# Modeling Turn-Taking with Semantically Informed Gestures

arXiv: https://arxiv.org/abs/2510.19350 · HTML: https://mlsanity.com/paper/2510.19350/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 48 references that are in the map.
- First posted on arXiv 22 Oct 2025.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis](https://mlsanity.com/paper/2203.05297/) (2022)
- [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023)
- [A Comprehensive Review of Data-Driven Co-Speech Gesture Generation](https://mlsanity.com/paper/2301.05339/) (2023)
- [TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog](https://mlsanity.com/paper/2010.10874/) (2020)
- [ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis](https://mlsanity.com/paper/2403.17936/) (2024)
- [Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis](https://mlsanity.com/paper/2412.06786/) (2024)
- [How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models](https://mlsanity.com/paper/2209.05161/) (2022)
- [Speech2Properties2Gestures: Gesture-Property Prediction as a Tool for Generating Representational Gestures from Speech](https://mlsanity.com/paper/2106.14736/) (2021)
- [Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals](https://mlsanity.com/paper/2505.12654/) (2025)
- [Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis](https://mlsanity.com/paper/2305.17971/) (2023)
- [Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction](https://mlsanity.com/paper/2505.21043/) (2025)
- [GestureCoach: Rehearsing for Engaging Talks with LLM-Driven Gesture Recommendations](https://mlsanity.com/paper/2504.10706/) (2025)
- [Enhancing Spoken Discourse Modeling in Language Models Using Gestural Cues](https://mlsanity.com/paper/2503.03474/) (2025)
- [Turn-taking annotation for quantitative and qualitative analyses of conversation](https://mlsanity.com/paper/2504.09980/) (2025)

## Built on by (most cited first, 1 of 1)

- [MIBURI: Towards Expressive Interactive Gesture Synthesis](https://mlsanity.com/paper/2603.03282/) (2026)

## Abstract (arXiv)

In conversation, humans use multimodal cues, such as speech, gestures, and gaze, to manage turn-taking. While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions. To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types. Using this dataset, we model turn-taking prediction through a Mixture-of-Experts framework integrating text, audio, and gestures. Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in multimodal turn-taking.