mlsanityData to 7 Oct 2026

Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders

arXiv 2607.07294 ↗First posted 8 Jul 2026cs.RO cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Turn-taking prediction is a key requirement for social robots involved in human-human interaction, particularly in mediator settings, where the robot must anticipate conversational dynamics rather than merely react to pauses. This work presents a Multimodal Voice Activity Projection (MM-VAP) framework that extends the original audio-only VAP formulation to synchronized audio-visual inputs while preserving its self-supervised future-projection objective. The proposed approach builds on pretrained audio-visual backbones originally optimized for speech-related tasks and adapts them through Low-Rank Adaptation to the multimodal turn-taking problem. After independent speaker encoding, an inter-speaker attention stage models the relational dynamics required to project future voice activity. In addition, a semantic consistency loss is introduced to regularize the 256-state output space according to higher-level dialogue activity patterns. Experiments on NoXi and NoXi+J showed improvements over the current baselines, particularly for some turn-taking events. Additional evaluation on the Haru EDR corpus further supported the suitability of this direction for mediation-oriented human-robot interaction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 42 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

19 of 42 references.

References · 19

  1. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 110
  2. 2022Voice Activity Projection: Self-supervised Learning of Turn-taking Eventscited by 54
  3. 2024LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASRcited by 35
  4. 2024Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translationcited by 35
  5. 2024Multilingual Turn-taking Prediction Using Voice Activity Projectioncited by 26
  6. 2024Real-time and Continuous Turn-taking Prediction Using Voice Activity Projectioncited by 21
  7. 2025Applying General Turn-taking Models to Conversational Human-Robot Interactioncited by 13
  8. 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detectioncited by 11
  9. 2025Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signalscited by 10
  10. 2024Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projectioncited by 8
  11. 2025Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interactioncited by 8
  12. 2024Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detectioncited by 6
7 more
  1. 2025A Noise-Robust Turn-Taking System for Real-World Dialogue Robots: A Field Experimentcited by 6
  2. 2024Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagementcited by 5
  3. 2025Triadic Multi-party Voice Activity Projection for Turn-taking in Spoken Dialogue Systemscited by 4
  4. 2025When and How to Express Empathy in Human-Robot Interaction Scenarioscited by 4
  5. 2025Voice Activity Projection Model with Multimodal Encoderscited by 2
  6. 2025Multimodal Transformer Models for Turn-taking Prediction: Effects on Conversational Dynamics of Human-Agent Interaction during Cooperative Gameplaycited by 1
  7. 2025M3SD: Multi-modal, Multi-scenario and Multi-language Speaker Diarization Datasetcited by 1

Built on by

Not cited yet.