# JoyVoice: Long-Context Conditioning for Anthropomorphic Multi-Speaker Conversational Synthesis

arXiv: https://arxiv.org/abs/2512.19090 · HTML: https://mlsanity.com/paper/2512.19090/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2025.

## Built on by (most cited first, 7 of 7)

- [MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning](https://mlsanity.com/paper/2601.01568/) (2026)
- [JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion](https://mlsanity.com/paper/2601.22143/) (2026)
- [Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis](https://mlsanity.com/paper/2601.13802/) (2026)
- [A2-LLM: An End-to-end Conversational Audio Avatar Large Language Model](https://mlsanity.com/paper/2602.04913/) (2026)
- [Tutti: Expressive Multi-Singer Synthesis via Structure-Level Timbre Control and Vocal Texture Modeling](https://mlsanity.com/paper/2602.08233/) (2026)
- [End-to-End Training for Discrete Token LLM based TTS System](https://mlsanity.com/paper/2606.09234/) (2026)
- [JoyAI-Voice 2.0: A Full-Continuous Autoregressive Speech Generation Model with Semantic-Acoustic Joint Representation](https://mlsanity.com/paper/2610.08834/) (2026)

## Abstract (arXiv)

Large speech generation models are evolving from single-speaker, short sentence synthesis to multi-speaker, long conversation geneartion. Current long-form speech generation models are predominately constrained to dyadic, turn-based interactions. To address this, we introduce JoyVoice, a novel anthropomorphic foundation model designed for flexible, boundary-free synthesis of up to eight speakers. Unlike conventional cascaded systems, JoyVoice employs a unified E2E-Transformer-DiT architecture that utilizes autoregressive hidden representations directly for diffusion inputs, enabling holistic end-to-end optimization. We further propose a MM-Tokenizer operating at a low bitrate of 12.5 Hz, which integrates multitask semantic and MMSE losses to effectively model both semantic and acoustic information. Additionally, the model incorporates robust text front-end processing via large-scale data perturbation. Experiments show that JoyVoice achieves state-of-the-art results in multilingual generation (Chinese, English, Japanese, Korean) and zero-shot voice cloning. JoyVoice achieves top-tier results on both the Seed-TTS-Eval Benchmark and multi-speaker long-form conversational voice cloning tasks, demonstrating superior audio quality and generalization. It achieves significant improvements in prosodic continuity for long-form speech, rhythm richness in multi-speaker conversations, paralinguistic naturalness, besides superior intelligibility. We encourage readers to listen to the demo at https://jea-speech.github.io/JoyVoice