# Multilingual and Continuous Backchannel Prediction: A Cross-lingual Study

arXiv: https://arxiv.org/abs/2512.14085 · HTML: https://mlsanity.com/paper/2512.14085/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 17 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Builds on (most cited first, 6 of 6)

- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Unsupervised pretraining transfers well across languages](https://mlsanity.com/paper/2002.02848/) (2020)
- [Voice Activity Projection: Self-supervised Learning of Turn-taking Events](https://mlsanity.com/paper/2205.09812/) (2022)
- [Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models](https://mlsanity.com/paper/2507.23159/) (2025)
- [Multilingual Turn-taking Prediction Using Voice Activity Projection](https://mlsanity.com/paper/2403.06487/) (2024)
- [Yeah, Right, Uh-Huh: A Deep Learning Backchannel Predictor](https://mlsanity.com/paper/1706.01340/) (2017)

## Built on by (most cited first, 1 of 1)

- [Controlling Backchannels in Streamable Full-duplex Models](https://mlsanity.com/paper/2609.29418/) (2026)

## Abstract (arXiv)

We present a multilingual, continuous backchannel prediction model for Japanese, English, and Chinese, and use it to investigate cross-linguistic timing behavior. The model is Transformer-based and operates at the frame level, jointly trained with auxiliary tasks on approximately 300 hours of dyadic conversations. Across all three languages, the multilingual model matches or surpasses monolingual baselines, indicating that it learns both language-universal cues and language-specific timing patterns. Zero-shot transfer with two-language training remains limited, underscoring substantive cross-lingual differences. Perturbation analyses reveal distinct cue usage: Japanese relies more on short-term linguistic information, whereas English and Chinese are more sensitive to silence duration and prosodic variation; multilingual training encourages shared yet adaptable representations and reduces overreliance on pitch in Chinese. A context-length study further shows that Japanese is relatively robust to shorter contexts, while Chinese benefits markedly from longer contexts. Finally, we integrate the trained model into a real-time processing software, demonstrating CPU-only inference. Together, these findings provide a unified model and empirical evidence for how backchannel timing differs across languages, informing the design of more natural, culturally-aware spoken dialogue systems.