# When can I Speak? Predicting initiation points for spoken dialogue agents

arXiv: https://arxiv.org/abs/2208.03812 · HTML: https://mlsanity.com/paper/2208.03812/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 29 references that are in the map.
- First posted on arXiv 7 Aug 2022.

## Builds on (most cited first, 5 of 5)

- [What is More Likely to Happen Next? Video-and-Language Future Event Prediction](https://mlsanity.com/paper/2010.07999/) (2020)
- [TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog](https://mlsanity.com/paper/2010.10874/) (2020)
- [Investigating Speech Features for Continuous Turn-Taking Prediction Using LSTMs](https://mlsanity.com/paper/1806.11461/) (2018)
- [Scalable Joint Models for Reliable Uncertainty-Aware Event Prediction](https://mlsanity.com/paper/1708.04757/) (2017)
- [Yeah, Right, Uh-Huh: A Deep Learning Backchannel Predictor](https://mlsanity.com/paper/1706.01340/) (2017)

## Built on by (most cited first, 9 of 9)

- [Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics](https://mlsanity.com/paper/2503.01174/) (2025)
- [Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction](https://mlsanity.com/paper/2505.21043/) (2025)
- [EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild](https://mlsanity.com/paper/2502.14892/) (2025)
- [Speculative End-Turn Detector for Efficient Speech Chatbot Assistant](https://mlsanity.com/paper/2503.23439/) (2025)
- [Visual Cues Support Robust Turn-taking Prediction in Noise](https://mlsanity.com/paper/2505.22088/) (2025)
- [Adaptive Endpointing with Deep Contextual Multi-armed Bandits](https://mlsanity.com/paper/2303.13407/) (2023)
- [Enabling Conversational Behavior Reasoning Capabilities in Full-Duplex Speech](https://mlsanity.com/paper/2512.21706/) (2025)
- [The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations](https://mlsanity.com/paper/2601.13835/) (2026)
- [S-MARC: Causal Streaming Reasoning for Full-Duplex Conversational Behavior Modeling](https://mlsanity.com/paper/2602.11065/) (2026)

## Abstract (arXiv)

Current spoken dialogue systems initiate their turns after a long period of silence (700-1000ms), which leads to little real-time feedback, sluggish responses, and an overall stilted conversational flow. Humans typically respond within 200ms and successfully predicting initiation points in advance would allow spoken dialogue agents to do the same. In this work, we predict the lead-time to initiation using prosodic features from a pre-trained speech representation model (wav2vec 1.0) operating on user audio and word features from a pre-trained language model (GPT-2) operating on incremental transcriptions. To evaluate errors, we propose two metrics w.r.t. predicted and true lead times. We train and evaluate the models on the Switchboard Corpus and find that our method outperforms features from prior work on both metrics and vastly outperforms the common approach of waiting for 700ms of silence.