mlsanityData to 14 Oct 2026

Realtime-Venus: A full-duplex interaction system with asynchronous delegation

arXiv 2609.13814 ↗First posted 12 Sep 2026cs.CV eess.AScited by 4

Not on the map. Why?

Abstract (arXiv)

Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requires grounding the conversation in evolving visual context. We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction. Each model serves as a complete conversational frontend, integrating continuous perception, conversational control, and native speech generation through a shared causal timeline for user inputs, model outputs, and delegation events. A dual-loop runtime coordinates live interaction with background reasoning and tool execution. Foreground interaction continues while Realtime-Venus-Harness executes tasks asynchronously and returns results for integration into the ongoing dialogue. Both models follow a common post-training recipe combining offline understanding, proactive full-duplex trajectories, and delegation workflows. Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%). Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81. On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech, respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 48 of its 60 references that are in the map. First posted on arXiv 12 Sep 2026.

Builds on

48 of 60 references.

References · 48

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2025Qwen3-VL Technical Reportcited by 2,793
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  6. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  8. 2025Qwen2.5-Omni Technical Reportcited by 736
  9. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  11. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  12. 2025Qwen3-Omni Technical Reportcited by 497
36 more
  1. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  2. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  3. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  4. 2025Kimi-Audio Technical Reportcited by 222
  5. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  6. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 144
  7. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  8. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 141
  9. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 117
  10. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  11. 2025Step-Audio 2 Technical Reportcited by 105
  12. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 103
  13. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 94
  14. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 85
  15. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 78
  16. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 65
  17. 2025MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligencecited by 61
  18. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  19. 2025Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Modelscited by 54
  20. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 47
  21. 2026Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluencycited by 37
  22. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 27
  23. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 25
  24. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 24
  25. 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 22
  26. 2025Fun-Audio-Chat Technical Reportcited by 19
  27. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 14
  28. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 12
  29. 2026OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understandingcited by 11
  30. 2026DuplexSLA: A Full-Duplex Spoken Language Model with Synchronized Speech, Language, and Actioncited by 9
  31. 2026Semantic-Aware Adaptive Visual Memory for Streaming Video Understandingcited by 8
  32. 2026DuplexOmni: Real-Time Listening, Seeing, Thinking, and Speaking for Full-Duplex Interactioncited by 8
  33. 2026AdaCodec: A Predictive Visual Code for Video MLLMscited by 7
  34. 2026UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Modelscited by 6
  35. 2026JoyAI-Talker: Full-Duplex Speech Interactive Large Model Built for Empathetic Voice Agentscited by 4
  36. 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videoscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.