mlsanityData to 7 Oct 2026

Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion

arXiv 2401.14717 ↗First posted 26 Jan 2024cs.CL cs.AI cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

We propose an approach for continuous prediction of turn-taking and backchanneling locations in spoken dialogue by fusing a neural acoustic model with a large language model (LLM). Experiments on the Switchboard human-human conversation dataset demonstrate that our approach consistently outperforms the baseline models with single modality. We also develop a novel multi-task instruction fine-tuning strategy to further benefit from LLM-encoded knowledge for understanding the tasks and conversational contexts, leading to additional improvements. Our approach demonstrates the potential of combined LLMs and acoustic models for a more natural and conversational interaction between humans and speech-enabled AI agents.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 12 of its 34 references that are in the map. First posted on arXiv 26 Jan 2024.

Builds on

12 of 34 references.

References · 12

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  5. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  6. 2020TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialogcited by 55
  7. 2022Voice Activity Projection: Self-supervised Learning of Turn-taking Eventscited by 54
  8. 2022Turn-Taking Prediction for Natural Conversational Speechcited by 15
  9. 2018Multimodal Continuous Turn-Taking Prediction Using Multiscale RNNscited by 12
  10. 2022Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialoguecited by 8
  11. 2018Improving End-of-turn Detection in Spoken Dialogues by Detecting Speaker Intentions as a Secondary Taskcited by 6
  12. 2023Oh, Jeez! or Uh-huh? A Listener-aware Backchannel Predictor on ASR Transcriptionscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 10.

20 papers.

Sort:

By area:

  1. 2025Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamicscited by 45
  2. 2024SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generationcited by 33
  3. 2025FLEXI: Benchmarking Full-duplex Human-LLM Speech Interactioncited by 14
  4. 2025Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signalscited by 10
  5. 2025ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systemscited by 7
  6. 2025Multi-Party Conversational Agents: A Surveycited by 6
  7. 2024Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagementcited by 5
  8. 2025Voice Activity Projection Model with Multimodal Encoderscited by 2
  9. 2026DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretrainingcited by 2
  10. 2024Device-Directed Speech Detection for Follow-up Conversations Using Large Language Modelscited by 1
  11. 2026Aligning Backchannel and Dialogue Context Representations via Contrastive LLM Fine-Tuningcited by 1
  12. 2026TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialoguecited by 1
8 more
  1. 2025Improving endpoint detection in end-to-end streaming ASR for conversational speechcited by 0
  2. 2026Sommelier: Scalable Open Multi-turn Audio Pre-processing for Full-duplex Speech Language Modelscited by 0
  3. 2026TokTalk: Expressive Real-time Facial Animation from Audio-LLM Tokenscited by 0
  4. 2026Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Frameworkcited by 0
  5. 2026Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialoguecited by 0
  6. 2026Using Semantic Uncertainty to Estimate Transition Relevance in Turn-takingcited by 0
  7. 2026Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projectioncited by 0
  8. 2026Voice-Light: A Full-Duplex Cascaded Voice Agent with Causal Turn-Taking and Speculative Generationcited by 0