# JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems

arXiv: https://arxiv.org/abs/2603.26515 · HTML: https://mlsanity.com/paper/2603.26515/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 27 Mar 2026.

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs](https://mlsanity.com/paper/2407.04051/) (2024)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Voice Activity Projection: Self-supervised Learning of Turn-taking Events](https://mlsanity.com/paper/2205.09812/) (2022)
- [OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation](https://mlsanity.com/paper/2410.17799/) (2024)
- [SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation](https://mlsanity.com/paper/2411.18138/) (2024)
- [Multilingual Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2403.06487/) (2024)
- [Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2401.04868/) (2024)
- [Applying General Turn-taking Models to Conversational Human-Robot Interaction](https://mlsanity.com/paper/2501.08946/) (2025)
- [Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systems](https://mlsanity.com/paper/2509.23938/) (2025)
- [HYBRIDFORMER: improving SqueezeFormer with hybrid attention and NSR mechanism](https://mlsanity.com/paper/2303.08636/) (2023)
- [ClapFM-EVC: High-Fidelity and Flexible Emotional Voice Conversion with Dual Control from Natural Language and Speech](https://mlsanity.com/paper/2505.13805/) (2025)
- [GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognition](https://mlsanity.com/paper/2405.02151/) (2024)

## Built on by (most cited first, 1 of 1)

- [X2-Turn: Frame-Synchronous Dual-Head Modeling for Joint Streaming ASR and Turn State Prediction](https://mlsanity.com/paper/2608.10878/) (2026)

## Abstract (arXiv)

Efficient and robust turn-taking detection remains a significant challenge in industrial-grade voice AI agents. Existing systems often rely on acoustic or linguistic cues alone, limiting accuracy and stability, whereas directly endowing large language models with full-duplex capability incurs heavy data and deployment costs that hinder real-time usage. We propose JAL-Turn, a lightweight speech-only turn-taking framework based on a joint acoustic-linguistic modeling paradigm, in which a cross-attention module adaptively fuses pre-trained acoustic and linguistic representations for low-latency hold or shift prediction. By sharing a frozen ASR encoder, JAL-Turn enables turn-taking prediction to run fully in parallel with speech recognition, introducing almost no additional end-to-end latency. We further introduce a scalable pipeline that automatically derives reliable turn-taking labels from large-scale real-world dialogue corpora. Experiments on the public Easy-Turn and Smart-Turn benchmarks and an in-house Japanese customer-service dataset show that JAL-Turn achieves competing results among strong baselines in detection accuracy while maintaining superior real-time performance.