mlsanityData to 14 Oct 2026

MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation

arXiv 2609.36903 ↗First posted 29 Sep 2026cs.CL cs.AI cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

End-to-end full-duplex speech models have brought open-source machine conversation closer to human-like interaction, yet existing systems remain limited in two intertwined dimensions: long-context robustness and multi-party interaction. Real-world scenarios such as meetings, group lessons, and social-robot reception require a single model to track, contextualize, and respond to multiple speakers over extended durations. Progress is constrained by both data and evaluation: open multi-party speech corpora remain small and are not designed for codec-frame-level full-duplex modeling, while existing long-audio benchmarks focus on passive listening and speech-to-speech benchmarks are mostly short and dyadic. We extend the Moshi paradigm jointly along the long-horizon and multi-party axes in English and Chinese. First, we release 57.6k hours of synthetic training data (hrefhttps://huggingface.co/datasets/MultiTalk/MultiTalkPTMultiTalkPT and hrefhttps://huggingface.co/datasets/MultiTalk/MultiTalkFTMultiTalkFT) for long-form, multi-party, English-Chinese full-duplex dialogue, with controllable length, participant count, turn-taking, overlap, backchannels, interruptions, addressee shifts, and long-range coreference. Second, we introduce hrefhttps://huggingface.co/datasets/MultiTalk/MultiTalkBenchMultiTalkBench, built from real human recordings, for evaluating long-form, multi-party, bilingual full-duplex dialogue. Conversations average 32.6 minutes and include probes for long-range entity tracking, topic coherence, and addressee selection. Third, we train a bilingual Moshi-style model that sustains coherent multi-party English-Chinese conversations over extended durations and substantially outperforms open-source baselines including Moshi, MiniCPM-o-4.5, and Qwen3-Omni-30B-A3B-Instruct on MultiTalkBench.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 56 of its 90 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

56 of 90 references.

References · 56

  1. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  2. 2025Qwen2.5-Omni Technical Reportcited by 736
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  4. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  5. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  6. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  7. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  8. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
  9. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 132
  10. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 100
  11. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 90
  12. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
38 more
  1. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  2. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  3. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 72
  4. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 72
  5. 2021M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challengecited by 71
  6. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 68
  7. 2023ICASSP 2023 Deep Noise Suppression Challengecited by 65
  8. 2021AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenariocited by 60
  9. 2022Voice Activity Projection: Self-supervised Learning of Turn-taking Eventscited by 60
  10. 2025Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Modelscited by 54
  11. 2024Scaling Speech-Text Pre-training with Synthetic Interleaved Datacited by 47
  12. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
  13. 2021ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversationcited by 44
  14. 2025Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examinercited by 41
  15. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
  16. 2021MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understandingcited by 36
  17. 2025MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoningcited by 35
  18. 2025URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Modelscited by 30
  19. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 27
  20. 2024Multilingual Turn-taking Prediction Using Voice Activity Projectioncited by 26
  21. 2025Not All Correct Answers Are Equal: Why Your Distillation Source Matterscited by 25
  22. 2025CodePlot-CoT: Mathematical Visual Reasoning by Thinking with Code-Driven Imagescited by 23
  23. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 22
  24. 2025MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Modelscited by 22
  25. 2017Addressee and Response Selection in Multi-Party Conversations with Speaker Interaction RNNscited by 21
  26. 2025FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systemscited by 20
  27. 2022TALCS: An Open-Source Mandarin-English Code-Switching Corpus and a Speech Recognition Baselinecited by 17
  28. 2025An LLM Benchmark for Addressee Recognition in Multi-modal Multi-party Dialoguecited by 15
  29. 2025WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learningcited by 15
  30. 2024ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generationcited by 14
  31. 2025MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocolscited by 14
  32. 2025FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Trainingcited by 11
  33. 2024Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversationscited by 9
  34. 2025LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encodingcited by 7
  35. 2024Empowering Character-level Text Infilling by Eliminating Sub-Tokenscited by 5
  36. 2025RoboEgo System Card: An Omnimodal Model with Native Full Duplexitycited by 5
  37. 2025Towards a Japanese Full-duplex Spoken Dialogue Systemcited by 5
  38. 2025Alignment with Fill-In-the-Middle for Enhancing Code Generationcited by 4

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.