# MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models

arXiv: https://arxiv.org/abs/2511.10262 · HTML: https://mlsanity.com/paper/2511.10262/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Builds on (most cited first, 26 of 26)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models](https://mlsanity.com/paper/2402.05044/) (2024)
- [GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot](https://mlsanity.com/paper/2412.02612/) (2024)
- [Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming](https://mlsanity.com/paper/2408.16725/) (2024)
- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM](https://mlsanity.com/paper/2411.00774/) (2024)
- [Spirit LM: Interleaved Spoken and Written Language Model](https://mlsanity.com/paper/2402.05755/) (2024)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities](https://mlsanity.com/paper/2503.04721/) (2025)
- [MinMo: A Multimodal Large Language Model for Seamless Voice Interaction](https://mlsanity.com/paper/2501.06282/) (2025)
- [Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction](https://mlsanity.com/paper/2502.17239/) (2025)
- [Language Model Can Listen While Speaking](https://mlsanity.com/paper/2408.02622/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models](https://mlsanity.com/paper/2507.23159/) (2025)
- [Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics](https://mlsanity.com/paper/2503.01174/) (2025)
- [Full-Duplex-Bench-v2: A Multi-Turn Evaluation Framework for Duplex Dialogue Systems with an Automated Examiner](https://mlsanity.com/paper/2510.07838/) (2025)
- [VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model](https://mlsanity.com/paper/2505.03739/) (2025)
- [CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models](https://mlsanity.com/paper/2501.14940/) (2025)
- [NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction](https://mlsanity.com/paper/2506.00975/) (2025)
- [VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models](https://mlsanity.com/paper/2501.04962/) (2025)
- [VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation](https://mlsanity.com/paper/2504.04060/) (2025)
- [Can Speech LLMs Think while Listening?](https://mlsanity.com/paper/2510.07497/) (2025)

## Built on by (most cited first, 22 of 22)

- [Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models](https://mlsanity.com/paper/2606.11167/) (2026)
- [Covo-Audio Technical Report](https://mlsanity.com/paper/2602.09823/) (2026)
- [Instruct-FD: Can Your Full-Duplex Speech System Follow Turn-Taking Instructions?](https://mlsanity.com/paper/2607.20460/) (2026)
- [IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows](https://mlsanity.com/paper/2606.19595/) (2026)
- [Overcoming State Inertia in Full-Duplex Spoken Language Models via Activation Steering](https://mlsanity.com/paper/2606.11386/) (2026)
- [MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents](https://mlsanity.com/paper/2609.24812/) (2026)
- [M3-DuplexBench: A Multi-Turn, Multilingual, Multidomain Benchmark for Full-Duplex Spoken Dialogue Models](https://mlsanity.com/paper/2607.29125/) (2026)
- [PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects](https://mlsanity.com/paper/2606.01016/) (2026)
- [DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents](https://mlsanity.com/paper/2609.03423/) (2026)
- [Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction](https://mlsanity.com/paper/2609.21392/) (2026)
- [MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation](https://mlsanity.com/paper/2609.36903/) (2026)
- [PACE: A Playback-Aligned Context Engine for LLM-Based Full-Duplex Voice Dialogue](https://mlsanity.com/paper/2608.07631/) (2026)
- [SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis](https://mlsanity.com/paper/2608.17931/) (2026)
- [DuplexJail: Safety Alignment Breaks Under Spoken Interruption in Full-Duplex Models](https://mlsanity.com/paper/2609.09420/) (2026)
- [Continue, Adapt, or Yield: In-Turn Adaptation to Overlapping Speech in Full-Duplex Agents](https://mlsanity.com/paper/2609.13117/) (2026)
- [RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue](https://mlsanity.com/paper/2609.16614/) (2026)
- [Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidance](https://mlsanity.com/paper/2609.30773/) (2026)
- [Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes](https://mlsanity.com/paper/2609.30798/) (2026)
- [Distributional Metrics for Evaluating Spoken Conversational Systems](https://mlsanity.com/paper/2609.31719/) (2026)
- [DuplexSpeechBench-Document Grounding: Benchmarking Document Grounding and Hallucinations in Voice Agents](https://mlsanity.com/paper/2610.00316/) (2026)
- [People Making Conversation: Evaluating How Naturally Speech Models Converse](https://mlsanity.com/paper/2610.12802/) (2026)
- [DuplexAct: How Interactive Are Frontier Full-Duplex Spoken Language Models?](https://mlsanity.com/paper/2610.17738/) (2026)

## Abstract (arXiv)

Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experience compared to traditional half-duplex models. However, existing benchmarks primarily focus on evaluating single-round interactions, neglecting the complexities of multi-round communication. Evaluating FD-SLMs in multi-round settings poses significant challenges, including blurred turn boundaries in communication and context inconsistency during model inference. Also, existing benchmarks often focus solely on evaluating conversational features, neglecting other critical aspects. To address these gaps, we introduce MTR-DuplexBench, a novel benchmark designed for a comprehensive multi-round evaluation of FD-SLMs. MTR-DuplexBench not only segments continuous full-duplex dialogues into discrete turns for turn-by-turn assessment but also incorporates various evaluation aspects, including conversational features, dialogue quality, instruction following, and safety. Experimental results reveal that current FD-SLMs face difficulties in maintaining consistent performance across multiple rounds and evaluation dimensions, highlighting the necessity and effectiveness of our benchmark. Code and data are available at: https://github.com/ZhangHe0918/MTR-DuplexBench