mlsanityData to 14 Oct 2026

Omni-Streaming Thinking

arXiv 2609.15128 ↗First posted 14 Sep 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Streaming omni-modal models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support an interpretation before an utterance or sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it. We call this failure premature cross-modal commitment. We propose Omni-Streaming Thinking (OST), which generates structured outputs that include evidence observed so far, forecasts of future evidence, and claims based on this evidence. Each claim is initially marked as pending and linked to a future verification interval. Audio and visual evidence are stored separately, and OST checks a claim against the evidence from the specified modality at the end of the verification interval. When contradictory evidence is detected, a refutation process reduces the influence of the claim and its dependent states, and then guides a state update using the new evidence. An answer gate decides whether the answer-critical claims meet the conditions for giving a response. Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, OST outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average. We also introduce OST-DiagBench, which holds video fixed and edits audio to test agreement, absence, contradiction, coexistence, and subtitle-speech conflict. OST reaches d-prime = 2.95, compared with at most 1.38 for open baselines, while reducing vision-induced auditory hallucinations.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 54 of its 62 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

54 of 62 references.

References · 54

  1. 2022Classifier-Free Diffusion Guidancecited by 5,847
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  4. 2025Qwen2.5-Omni Technical Reportcited by 736
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  7. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  8. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  9. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  11. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  12. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
38 more
  1. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  2. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  3. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  4. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153
  5. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 144
  6. 2017S3FD: Single Shot Scale-invariant Face Detectorcited by 142
  7. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  8. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 118
  9. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 113
  10. 2023Powerset multi-class cross entropy loss for neural speaker diarizationcited by 111
  11. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  12. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  13. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  14. 2023Stay on topic with Classifier-Free Guidancecited by 91
  15. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 88
  16. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 86
  17. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
  18. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 75
  19. 2025Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledgecited by 73
  20. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  21. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  22. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  23. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 43
  24. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 41
  25. 2025MLLMs are Deeply Affected by Modality Biascited by 40
  26. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 37
  27. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 22
  28. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 19
  29. 2024Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answeringcited by 18
  30. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  31. 2026OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understandingcited by 11
  32. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  33. 2025When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?cited by 8
  34. 2026Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Searchcited by 8
  35. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Modelscited by 8
  36. 2025AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimizationcited by 6
  37. 2026Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Modelscited by 6
  38. 2026Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisionscited by 5

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.