# Visual Cues Support Robust Turn-taking Prediction in Noise

arXiv: https://arxiv.org/abs/2505.22088 · HTML: https://mlsanity.com/paper/2505.22088/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 14 of 14)

- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Robust Self-Supervised Audio-Visual Speech Recognition](https://mlsanity.com/paper/2201.01763/) (2022)
- [Voice Activity Projection: Self-supervised Learning of Turn-taking Events](https://mlsanity.com/paper/2205.09812/) (2022)
- [Multilingual Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2403.06487/) (2024)
- [How Much Does Prosody Help Turn-taking? Investigations using Voice Activity Projection Models](https://mlsanity.com/paper/2209.05161/) (2022)
- [Investigating Speech Features for Continuous Turn-Taking Prediction Using LSTMs](https://mlsanity.com/paper/1806.11461/) (2018)
- [When can I Speak? Predicting initiation points for spoken dialogue agents](https://mlsanity.com/paper/2208.03812/) (2022)
- [How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition](https://mlsanity.com/paper/2004.08250/) (2020)
- [Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesis](https://mlsanity.com/paper/2305.17971/) (2023)
- [Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction](https://mlsanity.com/paper/2505.21043/) (2025)
- [Uncovering the Visual Contribution in Audio-Visual Speech Recognition](https://mlsanity.com/paper/2412.17129/) (2024)
- [Target conversation extraction: Source separation using turn-taking dynamics](https://mlsanity.com/paper/2407.11277/) (2024)
- [Neural Generation of Dialogue Response Timings](https://mlsanity.com/paper/2005.09128/) (2020)

## Built on by (most cited first, 3 of 3)

- [The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations](https://mlsanity.com/paper/2601.13835/) (2026)
- [Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection](https://mlsanity.com/paper/2609.14666/) (2026)
- [Audio-Visual Turn-taking Prediction in Cocktail Party Scenarios](https://mlsanity.com/paper/2609.17056/) (2026)

## Abstract (arXiv)

Accurate predictive turn-taking models (PTTMs) are essential for naturalistic human-robot interaction. However, little is known about their performance in noise. This study therefore explores PTTM performance in types of noise likely to be encountered once deployed. Our analyses reveal PTTMs are highly sensitive to noise. Hold/shift accuracy drops from 84% in clean speech to just 52% in 10 dB music noise. Training with noisy data enables a multimodal PTTM, which includes visual features to better exploit visual cues, with 72% accuracy in 10 dB music noise. The multimodal PTTM outperforms the audio-only PTTM across all noise types and SNRs, highlighting its ability to exploit visual cues; however, this does not always generalise to new types of noise. Analysis also reveals that successful training relies on accurate transcription, limiting the use of ASR-derived transcriptions to clean conditions. We make code publicly available for future research.