OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention
On the map: LLM reasoning → Audio-language model reasoning
Abstract (arXiv)
While humans perceive the world through diverse modalities that operate synergistically to support a holistic understanding of their surroundings, existing omnivideo models still face substantial challenges on audio-visual understanding tasks. In this paper, we propose OmniVideo-R1, a novel reinforced framework that improves mixed-modality reasoning. OmniVideo-R1 empowers models to "think with omnimodal cues" by two key strategies: (1) query-intensive grounding based on self-supervised learning paradigms; and (2) modality-attentive fusion built upon contrastive learning paradigms. Extensive experiments on multiple benchmarks demonstrate that OmniVideo-R1 consistently outperforms strong baselines, highlighting its effectiveness and robust generalization capabilities.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 72 references that are in the map. First posted on arXiv 5 Feb 2026.
Builds on
44 of 72 references.
Closest ancestors · 4
Same direction, newest first
- 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs12 Oct
- 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities23 May
- 2025EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning7 May
- 2025Qwen2.5-Omni Technical Report26 Mar
Other references · 40
- 2025Qwen3 Technical Reportcited by 8,195
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
- 2025Group Sequence Policy Optimizationcited by 788
- 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
- 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
- 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
28 more
- 2025Qwen3-Omni Technical Reportcited by 451
- 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
- 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
- 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
- 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
- 2025GRIT: Teaching MLLMs to Think with Imagescited by 121
- 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
- 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 107
- 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
- 2025R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learningcited by 99
- 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
- 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
- 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 58
- 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 58
- 2025HumanOmniV2: From Understanding to Omni-Modal Reasoning with Contextcited by 50
- 2024Baichuan-Omni Technical Reportcited by 42
- 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
- 2025VisRL: Intention-Driven Visual Perception via Reinforced Reasoningcited by 39
- 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
- 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 38
- 2025ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shortscited by 36
- 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learningcited by 32
- 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 31
- 2025ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Scene Reconstructioncited by 27
- 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoningcited by 25
- 2025Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, and LLaMAcited by 23
- 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 14
- 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
Built on by
Citing papers per year: 2026: 9.
9 papers.
- 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 21
- 2026SEAL: Synergistic Co-Evolution of Agents and Learning Environmentscited by 9
- 2026Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Groundingcited by 6
- 2026When Thinking Hurts: Mitigating Visual Forgetting in Video Reasoning via Frame Repetitioncited by 3
- 20264DThinker: Thinking with 4D Imagery for Dynamic Spatial Understandingcited by 3
- 2026OmniReasoner: Thinking with Long Audio-Video via Native Tool Usecited by 1
- 2026OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reorderingcited by 0
- 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 0
- 2026Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMscited by 0
- 2026Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs29 Sep 2026
- 2026OmniReasoner: Thinking with Long Audio-Video via Native Tool Use21 Jul 2026
- 2026SEAL: Synergistic Co-Evolution of Agents and Learning Environments23 May 2026
- 20264DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding7 May 2026
- 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs16 Apr 2026
- 2026Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Grounding13 Apr 2026
- 2026OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering9 Apr 2026
- 2026When Thinking Hurts: Mitigating Visual Forgetting in Video Reasoning via Frame Repetition17 Mar 2026
- 2026OmniGAIA: Towards Native Omni-Modal AI Agents26 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2026LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning
- 2026OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning
- 2026Omni-o3: Deep Nested Omnimodal Deduction for Deliberative Audio-Visual Reasoning
- 2026AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers
- 2025Active Perception Agent for Omnimodal Audio-Video Understanding
- 2026Empowering Long-form Omni-modal Understanding with Robust Audio Perception
- 2026Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning
- 2026Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought