mlsanityData to 14 Oct 2026

Multimodal Duplex Interaction Agent

arXiv 2609.08977 ↗First posted 8 Sep 2026eess.AS cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In this work, we present Gander, a native multimodal duplex interaction model that builds on MiniCPM-o 4.5 and is further adapted for realtime interaction with an asynchronous agent loop. In contrast to conventional turn based systems, Gander continuously processes streaming user inputs, enabling full-duplex interaction in both everyday conversations and complex workflow agent scenarios. Users can interrupt an ongoing response, while the model can proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) a Cerebellum-Brain collaborative framework, Cerebellum is responsible for realtime interaction while the Brain handles complex reasoning and higher level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, where user inputs and model outputs are flattened into an ordered token stream at the chunk level. We evaluate Gander across conversational ability, interactive capability, understanding, and tool assisted task execution. Internal human evaluations show that Gander maintains natural and expressive spoken dialogue, while benchmark results demonstrate effective turn taking capability and encouraging results on spoken question answering and related understanding tasks. Gander also supports a range of challenging interaction settings, including background noise interference, multi-party interactions, and backchannel communication. While our current evaluation focuses on tool assisted settings, broader long horizon agent tasks and more diverse deployment conditions remain promising directions for further study. We release Gander together with its models, code, and data to facilitate further research and development in the community.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 54 of its 67 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

54 of 67 references.

References · 54

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  8. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  9. 2025OpenAI GPT-5 System Cardcited by 989
  10. 2025Qwen2.5-Omni Technical Reportcited by 736
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  12. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
38 more
  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  2. 2024Qwen2-Audio Technical Reportcited by 560
  3. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  4. 2025Qwen3-Omni Technical Reportcited by 497
  5. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  6. 2025Kimi-Audio Technical Reportcited by 222
  7. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  8. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  9. 2026Qwen3.5-Omni Technical Reportcited by 174
  10. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  11. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  12. 2025CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-trainingcited by 150
  13. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 141
  14. 2026Kimi K3: Open Frontier Intelligencecited by 138
  15. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  16. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 122
  17. 2026Qwen3-TTS Technical Reportcited by 118
  18. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  19. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  20. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 109
  21. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  22. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
  23. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 78
  24. 2025Streaming Video Instruction Tuningcited by 26
  25. 2025Fun-ASR Technical Reportcited by 19
  26. 2024ControlSpeech: Towards Simultaneous and Independent Zero-shot Speaker Cloning and Zero-shot Language Style Controlcited by 17
  27. 2026FireRedASR2S: A State-of-the-Art Industrial-Grade All-in-One Automatic Speech Recognition Systemcited by 16
  28. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 16
  29. 2024MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speechcited by 15
  30. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  31. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 14
  32. 2025LiveCC: Learning Video LLM with Streaming Speech Transcription at Scalecited by 9
  33. 2025Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Modelscited by 9
  34. 2026DuplexOmni: Real-Time Listening, Seeing, Thinking, and Speaking for Full-Duplex Interactioncited by 8
  35. 2026A Survey of Full-Duplex Spoken Dialogue Systems: Architectural Hierarchy, Interaction Ontology, and Decision State Machinecited by 8
  36. 2024Language-Codec: Bridging Discrete Codec Representations and Speech Language Modelscited by 6
  37. 2026BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLMcited by 6
  38. 2026Audio Interaction Modelcited by 4

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions
  2. 2024Enabling Real-Time Conversations with Minimal Training Costs
  3. 2026BrainAgent: A Large Language Model-Driven Multi-Agent Framework for Autonomous Brain Signal Understanding
  4. 2025OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds
  5. 2026τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains
  6. 2026U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation
  7. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
  8. 2026DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoning