mlsanityData to 7 Oct 2026

video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model

arXiv 2502.11775 ↗First posted 17 Feb 2025cs.CVcited by 19

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

While recent advancements in reasoning optimization have significantly enhanced the capabilities of large language models (LLMs), existing efforts to improve reasoning have been limited to solving mathematical problems and focusing on visual graphical inputs, neglecting broader applications in general video understanding.This paper proposes video-SALMONN-o1, the first open-source reasoning-enhanced audio-visual LLM designed for general video understanding tasks. To enhance its reasoning abilities, we develop a reasoning-intensive dataset featuring challenging audio-visual questions with step-by-step solutions. We also propose process direct preference optimization (pDPO), which leverages contrastive step selection to achieve efficient step-level reward modelling tailored for multimodal inputs. Additionally, we introduce RivaBench, the first reasoning-intensive video understanding benchmark, featuring over 4,000 high-quality, expert-curated question-answer pairs across scenarios such as standup comedy, academic presentations, and synthetic video detection. video-SALMONN-o1 achieves 3-8% accuracy improvements over the LLaVA-OneVision baseline across different video reasoning benchmarks. Besides, pDPO achieves 6-8% improvements compared to the supervised fine-tuning model on RivaBench. Enhanced reasoning enables video-SALMONN-o1 zero-shot synthetic video detection capabilities.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 63 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

55 of 63 references.

References · 55

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
38 more
  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  4. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  5. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  7. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  8. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  9. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  11. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  12. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  13. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  14. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  15. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  16. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  17. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  18. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  19. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  20. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  21. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  22. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  23. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  24. 2019Audio-Visual Scene-Aware Dialogcited by 122
  25. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 117
  26. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  27. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  28. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  29. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  30. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  31. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  32. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  33. 2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 84
  34. 2024o1-Coder: an o1 Replication for Codingcited by 77
  35. 2025Virgo: A Preliminary Exploration on Reproducing o1-like MLLMcited by 68
  36. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  37. 2023FunQA: Towards Surprising Video Comprehensioncited by 35
  38. 2023SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Modelscited by 28

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  2. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  3. 2025Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learningcited by 37
  4. 2025ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shortscited by 36
  5. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 29
  6. 2025Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learningcited by 19
  7. 2025ALLM4ADD: Unlocking the Capabilities of Audio Large Language Models for Audio Deepfake Detectioncited by 17
  8. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 13
  9. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 9
  10. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  11. 2026TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captionscited by 8
  12. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
7 more
  1. 2026D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioningcited by 6
  2. 2025video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLMcited by 4
  3. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMscited by 4
  4. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  5. 2026Uni-Synergy: Bridging Understanding and Generation for Personalized Reasoning via Co-operative Reinforcement Learningcited by 0
  6. 2026SLVMBench: Skill Learning from Video Memorycited by 0
  7. 2026OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
  2. 2025Audio-centric Video Understanding Benchmark without Text Shortcut
  3. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
  4. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs
  5. 2023Audio-Visual LLM for Video Understanding
  6. 2025R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarios
  7. 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos
  8. 2025SiLVR: A Simple Language-based Video Reasoning Framework