Multimodal Duplex Interaction Agent
Not on the map. Why?
Abstract (arXiv)
In this work, we present Gander, a native multimodal duplex interaction model that builds on MiniCPM-o 4.5 and is further adapted for realtime interaction with an asynchronous agent loop. In contrast to conventional turn based systems, Gander continuously processes streaming user inputs, enabling full-duplex interaction in both everyday conversations and complex workflow agent scenarios. Users can interrupt an ongoing response, while the model can proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) a Cerebellum-Brain collaborative framework, Cerebellum is responsible for realtime interaction while the Brain handles complex reasoning and higher level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, where user inputs and model outputs are flattened into an ordered token stream at the chunk level. We evaluate Gander across conversational ability, interactive capability, understanding, and tool assisted task execution. Internal human evaluations show that Gander maintains natural and expressive spoken dialogue, while benchmark results demonstrate effective turn taking capability and encouraging results on spoken question answering and related understanding tasks. Gander also supports a range of challenging interaction settings, including background noise interference, multi-party interactions, and backchannel communication. While our current evaluation focuses on tool assisted settings, broader long horizon agent tasks and more diverse deployment conditions remain promising directions for further study. We release Gander together with its models, code, and data to facilitate further research and development in the community.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 54 of its 67 references that are in the map. First posted on arXiv 8 Sep 2026.
Builds on
54 of 67 references.
References · 54
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2024GPT-4o System Cardcited by 4,588
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
- 2025OpenAI GPT-5 System Cardcited by 989
- 2025Qwen2.5-Omni Technical Reportcited by 736
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
- 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
38 more
- 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
- 2024Qwen2-Audio Technical Reportcited by 560
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
- 2025Qwen3-Omni Technical Reportcited by 497
- 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
- 2025Kimi-Audio Technical Reportcited by 222
- 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
- 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
- 2026Qwen3.5-Omni Technical Reportcited by 174
- 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
- 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
- 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
- 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 141
- 2026Kimi K3: Open Frontier Intelligencecited by 138
- 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
- 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
- 2026Qwen3-TTS Technical Reportcited by 118
- 2025Baichuan-Omni-1.5 Technical Reportcited by 111
- 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
- 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 109
- 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
- 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
- 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 78
- 2025Streaming Video Instruction Tuningcited by 26
- 2025Fun-ASR Technical Reportcited by 19
- 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Controlcited by 17
- 2026FireRedASR2S: A State-of-the-Art Industrial-Grade All-in-One Automatic Speech Recognition Systemcited by 16
- 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 16
- 2024MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speechcited by 15
- 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
- 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 14
- 2025LiveCC: Learning Video LLM with Streaming Speech Transcription at Scalecited by 9
- 2025Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Modelscited by 9
- 2026DuplexOmni: Real-Time Listening, Seeing, Thinking, and Speaking for Full-Duplex Interactioncited by 8
- 2026A Survey of Full-Duplex Spoken Dialogue Systems: Architectural Hierarchy, Interaction Ontology, and Decision State Machinecited by 8
- 2024Language-Codec: Bridging Discrete Codec Representations and Speech Language Modelscited by 6
- 2026BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLMcited by 6
- 2026Audio Interaction Modelcited by 4
50 most cited of 54 All 54 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agentscited by 1
- 2026When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipelinecited by 0
- 2026DuplexAgent-RSI: Recursive Harness Improvement for Full-Duplex Voice Agent Collaborationcited by 0
- 2026DuplexAgent-RSI: Recursive Harness Improvement for Full-Duplex Voice Agent Collaboration8 Oct 2026
- 2026SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents28 Sep 2026
- 2026When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline10 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions
- 2024Enabling Real-Time Conversations with Minimal Training Costs
- 2026BrainAgent: A Large Language Model-Driven Multi-Agent Framework for Autonomous Brain Signal Understanding
- 2025OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds
- 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains
- 2026U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation
- 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
- 2026DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoning