# Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion

arXiv: https://arxiv.org/abs/2401.14717 · HTML: https://mlsanity.com/paper/2401.14717/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 34 references that are in the map.
- First posted on arXiv 26 Jan 2024.

## Builds on (most cited first, 12 of 12)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Jigsaw: Large Language Models meet Program Synthesis](https://mlsanity.com/paper/2112.02969/) (2021)
- [Voice Activity Projection: Self-supervised Learning of Turn-taking Events](https://mlsanity.com/paper/2205.09812/) (2022)
- [TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog](https://mlsanity.com/paper/2010.10874/) (2020)
- [Turn-Taking Prediction for Natural Conversational Speech](https://mlsanity.com/paper/2208.13321/) (2022)
- [Multimodal Continuous Turn-Taking Prediction Using Multiscale RNNs](https://mlsanity.com/paper/1808.10785/) (2018)
- [Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue](https://mlsanity.com/paper/2204.10172/) (2022)
- [Improving End-of-turn Detection in Spoken Dialogues by Detecting Speaker Intentions as a Secondary Task](https://mlsanity.com/paper/1805.06511/) (2018)
- [Oh, Jeez! or Uh-huh? A Listener-aware Backchannel Predictor on ASR Transcriptions](https://mlsanity.com/paper/2304.04478/) (2023)

## Built on by (most cited first, 20 of 20)

- [Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics](https://mlsanity.com/paper/2503.01174/) (2025)
- [SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation](https://mlsanity.com/paper/2411.18138/) (2024)
- [FLEXI: Benchmarking Full-duplex Human-LLM Speech Interaction](https://mlsanity.com/paper/2509.22243/) (2025)
- [Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals](https://mlsanity.com/paper/2505.12654/) (2025)
- [ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems](https://mlsanity.com/paper/2503.08533/) (2025)
- [Multi-Party Conversational Agents: A Survey](https://mlsanity.com/paper/2505.18845/) (2025)
- [Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagement](https://mlsanity.com/paper/2409.13726/) (2024)
- [DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining](https://mlsanity.com/paper/2603.08216/) (2026)
- [Voice Activity Projection Model with Multimodal Encoders](https://mlsanity.com/paper/2506.03980/) (2025)
- [Device-Directed Speech Detection for Follow-up Conversations Using Large Language Models](https://mlsanity.com/paper/2411.00023/) (2024)
- [Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuning](https://mlsanity.com/paper/2604.16622/) (2026)
- [TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue](https://mlsanity.com/paper/2607.01345/) (2026)
- [Improving endpoint detection in end-to-end streaming ASR for conversational speech](https://mlsanity.com/paper/2505.17070/) (2025)
- [Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Models](https://mlsanity.com/paper/2603.25750/) (2026)
- [TokTalk: Expressive Real-time Facial Animation from Audio-LLM Tokens](https://mlsanity.com/paper/2605.31294/) (2026)
- [Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework](https://mlsanity.com/paper/2608.28630/) (2026)
- [Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialogue](https://mlsanity.com/paper/2609.03321/) (2026)
- [Using Semantic Uncertainty to Estimate Transition Relevance in Turn-taking](https://mlsanity.com/paper/2609.10934/) (2026)
- [Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection](https://mlsanity.com/paper/2609.14666/) (2026)
- [Voice-Light: A Full-Duplex Cascaded Voice Agent with Causal Turn-Taking and Speculative Generation](https://mlsanity.com/paper/2609.20995/) (2026)

## Abstract (arXiv)

We propose an approach for continuous prediction of turn-taking and backchanneling locations in spoken dialogue by fusing a neural acoustic model with a large language model (LLM). Experiments on the Switchboard human-human conversation dataset demonstrate that our approach consistently outperforms the baseline models with single modality. We also develop a novel multi-task instruction fine-tuning strategy to further benefit from LLM-encoded knowledge for understanding the tasks and conversational contexts, leading to additional improvements. Our approach demonstrates the potential of combined LLMs and acoustic models for a more natural and conversational interaction between humans and speech-enabled AI agents.