# Investigating Speech Features for Continuous Turn-Taking Prediction Using LSTMs

arXiv: https://arxiv.org/abs/1806.11461 · HTML: https://mlsanity.com/paper/1806.11461/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2018.

## Built on by (most cited first, 16 of 16)

- [Generative Spoken Dialogue Language Modeling](https://mlsanity.com/paper/2203.16502/) (2022)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [When can I Speak? Predicting initiation points for spoken dialogue agents](https://mlsanity.com/paper/2208.03812/) (2022)
- [Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue](https://mlsanity.com/paper/2204.10172/) (2022)
- [Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction](https://mlsanity.com/paper/2505.21043/) (2025)
- [What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Study](https://mlsanity.com/paper/2109.09105/) (2021)
- [Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systems](https://mlsanity.com/paper/2510.02066/) (2025)
- [Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2601.19063/) (2026)
- [DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining](https://mlsanity.com/paper/2603.08216/) (2026)
- [Visual Cues Support Robust Turn-taking Prediction in Noise](https://mlsanity.com/paper/2505.22088/) (2025)
- [Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Models](https://mlsanity.com/paper/2501.00953/) (2025)
- [NaturalTurn: A Method to Segment Speech into Psychologically Meaningful Conversational Turns](https://mlsanity.com/paper/2403.15615/) (2024)
- [Multimodal Transformer Models for Turn-taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction during Cooperative Gameplay](https://mlsanity.com/paper/2503.16432/) (2025)
- [The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations](https://mlsanity.com/paper/2601.13835/) (2026)
- [Low-Latency Turn-Taking via Context-Aware Preface Generation in a Real-World Dialogue Robot](https://mlsanity.com/paper/2607.23204/) (2026)
- [BanglaTurn: A Benchmark and Whisper-Based Model for End-of-Turn Detection in Bangla Speech](https://mlsanity.com/paper/2609.29371/) (2026)

## Abstract (arXiv)

For spoken dialog systems to conduct fluid conversational interactions with users, the systems must be sensitive to turn-taking cues produced by a user. Models should be designed so that effective decisions can be made as to when it is appropriate, or not, for the system to speak. Traditional end-of-turn models, where decisions are made at utterance end-points, are limited in their ability to model fast turn-switches and overlap. A more flexible approach is to model turn-taking in a continuous manner using RNNs, where the system predicts speech probability scores for discrete frames within a future window. The continuous predictions represent generalized turn-taking behaviors observed in the training data and can be applied to make decisions that are not just limited to end-of-turn detection. In this paper, we investigate optimal speech-related feature sets for making predictions at pauses and overlaps in conversation. We find that while traditional acoustic features perform well, part-of-speech features generally perform worse than word features. We show that our current models outperform previously reported baselines.