Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks also fits: Multimodal and embodied agents
Abstract (arXiv)
Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M2-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns whether, when, and how the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of τ-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 23 of its 28 references that are in the map. First posted on arXiv 21 Sep 2026.
Builds on
23 of 28 references.
Closest ancestors · 2
Same direction, newest first
- 2026Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency6 Apr
- 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains14 Mar
Other references · 21
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
- 2015Policy Distillationcited by 414
- 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
- 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
- 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
- 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 171
- 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluationcited by 139
- 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
- 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
- 2023Evaluating Hallucinations in Chinese Large Language Modelscited by 33
9 more
- 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 18
- 2025Fun-Audio-Chat Technical Reportcited by 17
- 2026EVA-Bench: A New End-to-end Framework for Evaluating Voice Agentscited by 16
- 2024A Chinese Dataset for Evaluating the Safeguards in Large Language Modelscited by 11
- 2025RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playingcited by 10
- 2025EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Modelscited by 10
- 2025MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Modelscited by 7
- 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representationscited by 6
- 2026Qwen-Audio-3.0-ASR Technical Reportcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes
- 2026SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning
- 2026SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents
- 2025Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents
- 2025VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing
- 2026Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use
- 2026Talk2Agent: Benchmarking Voice Interfaces for Text Agents
- 2026LTS-VoiceAgent: A Listen-Think-Speak Framework for Efficient Streaming Voice Interaction via Semantic Triggering and Incremental Reasoning