# Towards a Japanese Full-duplex Spoken Dialogue System

arXiv: https://arxiv.org/abs/2506.02979 · HTML: https://mlsanity.com/paper/2506.02979/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 19 of 19)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [LLaMA-Omni: Seamless Speech Interaction with Large Language Models](https://mlsanity.com/paper/2409.06666/) (2024)
- [Generative Spoken Dialogue Language Modeling](https://mlsanity.com/paper/2203.16502/) (2022)
- [Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents](https://mlsanity.com/paper/2409.15594/) (2024)
- [Language Model Can Listen While Speaking](https://mlsanity.com/paper/2408.02622/) (2024)
- [OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation](https://mlsanity.com/paper/2410.17799/) (2024)
- [A Full-duplex Speech Dialogue Scheme Based On Large Language Models](https://mlsanity.com/paper/2405.19487/) (2024)
- [SEANet: A Multi-modal Speech Enhancement Network](https://mlsanity.com/paper/2009.02095/) (2020)
- [SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation](https://mlsanity.com/paper/2411.18138/) (2024)
- [Empirical Analysis of Training Strategies of Transformer-based Japanese Chit-chat Systems](https://mlsanity.com/paper/2109.05217/) (2021)
- [J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modeling](https://mlsanity.com/paper/2407.15828/) (2024)

## Built on by (most cited first, 5 of 5)

- [From Turn-Taking to Synchronous Dialogue: A Survey of Full-Duplex Spoken Language Models](https://mlsanity.com/paper/2509.14515/) (2025)
- [M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models](https://mlsanity.com/paper/2607.29125/) (2026)
- [Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems](https://mlsanity.com/paper/2606.21970/) (2026)
- [MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation](https://mlsanity.com/paper/2609.36903/) (2026)
- [IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systems](https://mlsanity.com/paper/2606.06559/) (2026)

## Abstract (arXiv)

Full-duplex spoken dialogue systems, which can model simultaneous bidirectional features of human conversations such as speech overlaps and backchannels, have attracted significant attention recently. However, the study of full-duplex spoken dialogue systems for the Japanese language has been limited, and the research on their development in Japanese remains scarce. In this paper, we present the first publicly available full-duplex spoken dialogue model in Japanese, which is built upon Moshi, a full-duplex dialogue model in English. Our model is trained through a two-stage process: pre-training on a large-scale spoken dialogue data in Japanese, followed by fine-tuning on high-quality stereo spoken dialogue data. We further enhance the model's performance by incorporating synthetic dialogue data generated by a multi-stream text-to-speech system. Evaluation experiments demonstrate that the trained model outperforms Japanese baseline models in both naturalness and meaningfulness.