mlsanityData to 7 Oct 2026

OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts

arXiv 2503.22952 ↗First posted 29 Mar 2025cs.CVcited by 48

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

The rapid advancement of multi-modal language models (MLLMs) like GPT-4o has propelled the development of Omni language models, designed to process and proactively respond to continuous streams of multi-modal data. Despite their potential, evaluating their real-world interactive capabilities in streaming video contexts remains a formidable challenge. In this work, we introduce OmniMMI, a comprehensive multi-modal interaction benchmark tailored for OmniLLMs in streaming video contexts. OmniMMI encompasses over 1,121 videos and 2,290 questions, addressing two critical yet underexplored challenges in existing video benchmarks: streaming video understanding and proactive reasoning, across six distinct subtasks. Moreover, we propose a novel framework, Multi-modal Multiplexing Modeling (M4), designed to enable an inference-efficient streaming model that can see, listen while generating.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 61 references that are in the map. First posted on arXiv 29 Mar 2025.

Builds on

49 of 61 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding6 Nov
  2. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding22 Sep
  3. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture4 Sep
  4. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos19 Aug
  5. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  6. 2024Long Context Transfer from Language to Vision24 Jun
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  8. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  9. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding8 Apr
  10. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens4 Apr
  11. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  12. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
1 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov

Other references · 36

  1. 2024GPT-4o System Cardcited by 4,467
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  7. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  8. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  9. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  10. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  11. 2024Qwen2-Audio Technical Reportcited by 543
  12. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
24 more
  1. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  2. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  3. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  5. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  6. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  7. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  8. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 204
  9. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  10. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  11. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  12. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  13. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  14. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  15. 2024Streaming Dense Video Captioningcited by 82
  16. 2024Language Model Can Listen While Speakingcited by 62
  17. 2023Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videoscited by 54
  18. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  19. 2024Grounded Multi-Hop VideoQA in Long-Form Egocentric Videoscited by 43
  20. 2024Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Modelscited by 26
  21. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  22. 2022DataMUX: Data Multiplexing for Neural Networkscited by 18
  23. 2023Streaming Video Modelcited by 10
  24. 2024Enabling Real-Time Conversations with Minimal Training Costscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 41.

48 papers.

Sort:

By area:

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  2. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  3. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  4. 2026A Simple Baseline for Streaming Video Understandingcited by 29
  5. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 24
  6. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 18
  7. 2026AURA: Always-On Understanding and Real-Time Assistance via Video Streamscited by 15
  8. 2026SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Modelscited by 12
  9. 2026PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarioscited by 10
  10. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  11. 2026What Should a Streaming Video Model Remember?cited by 7
  12. 2026ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understandingcited by 5
36 more
  1. 2026MOSS-VL Technical Reportcited by 5
  2. 2026LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Taskscited by 4
  3. 2026IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streamscited by 4
  4. 2026StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Videocited by 3
  5. 2026OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistantscited by 3
  6. 2026EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Visioncited by 3
  7. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  8. 2026Harnessing Streaming Video in the Wildcited by 3
  9. 2026FOLIO: Focused Semantic Memory for Streaming Video Understandingcited by 3
  10. 2026StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understandingcited by 3
  11. 2026StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMscited by 3
  12. 2025Chat with AI: The Surprising Turn of Real-time Video Communication from Human to AIcited by 2
  13. 2025Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?cited by 2
  14. 2026Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interactioncited by 2
  15. 2026Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoningcited by 2
  16. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 2
  17. 2026HumanOmni-Speaker: Identifying Who said What and Whencited by 1
  18. 2026EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streamscited by 1
  19. 2026TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videoscited by 1
  20. 2026VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agentscited by 1
  21. 2026PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warningcited by 1
  22. 2026GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Videocited by 1
  23. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  24. 2026Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interactioncited by 1
  25. 2025Temporally-Grounded Language Generation: Towards Real-Time Vision-Language Modelscited by 0
  26. 2026Artic: AI-oriented Real-time Communication for MLLM Video Assistantcited by 0
  27. 2026SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videoscited by 0
  28. 2026X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understandingcited by 0
  29. 2026OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMscited by 0
  30. 2026Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni modelscited by 0
  31. 2026LiveProBench: Can Streaming Video Models Really Interact Like Humans?cited by 0
  32. 2026OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialoguecited by 0
  33. 2026Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streamscited by 0
  34. 2026TRACE: Temporal Audit and Condition-aware Evaluation of Streaming Video Understandingcited by 0
  35. 2026Agentic Video Understanding: A Surveycited by 0
  36. 2026OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction
  2. 2026StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering
  3. 2026OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming
  4. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions
  5. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models
  6. 2024StreamChat: Chatting with Streaming Video
  7. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
  8. 2026LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understanding