mlsanityData to 14 Oct 2026

video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models

arXiv 2506.15220 ↗First posted 18 Jun 2025cs.CV cs.CL cs.SDcited by 47

Not on the map. Why?

Abstract (arXiv)

We present video-SALMONN 2, a family of audio-visual large language models that set new state-of-the-art (SOTA) results in video description and question answering (QA). Our core contribution is multi-round direct preference optimisation (MrDPO), paired with a caption-quality objective that jointly rewards completeness and factual accuracy. Unlike standard DPO with a fixed reference policy, MrDPO periodically refreshes the reference by bootstrapping from a newly re-initialised lightweight adapter trained on the latest preferences, avoiding reference staleness and enabling continual improvement. This strategy produces captions that are consistently more detailed and accurate than those from proprietary systems such as GPT-4o and Gemini-1.5 Pro. We further distil these gains by using our model to generate a high-quality video-caption corpus for supervised fine-tuning of new models, transferring benefits beyond captioning to strong performance on complex video-QA tasks. Across widely used audio-visual and visual-only understanding benchmarks (including Video-MME, WorldSense, AVUT, Video-Holmes, DailyOmni, MLVU, and LVBench), our 3B and 7B models achieve SOTA results at comparable scales, while the 72B model surpasses all other open-source systems. Our source code, models, and data are released at hrefhttps://github.com/bytedance/video-SALMONN-2https://github.com/bytedance/video-SALMONN-2.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 51 of its 62 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

51 of 62 references.

References · 51

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2025Qwen2.5-VL Technical Reportcited by 5,587
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  12. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  9. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  10. 2025Qwen2.5-Omni Technical Reportcited by 736
  11. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  13. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  14. 2024Qwen2-Audio Technical Reportcited by 560
  15. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  16. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  17. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  18. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  19. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  20. 2024Iterative Reasoning Preference Optimizationcited by 246
  21. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  22. 2023Listen, Think, and Understandcited by 209
  23. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  24. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  25. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  26. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  27. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  28. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  29. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  30. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 118
  31. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 114
  32. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  33. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  34. 2023Joint Audio and Speech Understandingcited by 100
  35. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 36
  36. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 36
  37. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  38. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 24

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 45.

47 papers.

Sort:

By area:

  1. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 17
  2. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 11
  3. 2026OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understandingcited by 11
  4. 2026Script-a-Video: Deep Structured Audio-visual Captions via Factorized Streams and Relational Groundingcited by 8
  5. 2026OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chainscited by 8
  6. 2026FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMscited by 7
  7. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6
  8. 2026OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Videocited by 6
  9. 2026Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Modelscited by 6
  10. 2025A Benchmark for Omni-Modal Reasoning in Long Videoscited by 5
  11. 2026AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimizationcited by 4
  12. 2026JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environmentscited by 4
35 more
  1. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  2. 2026Realtime-Venus: A full-duplex interaction system with asynchronous delegationcited by 4
  3. 2026Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMscited by 3
  4. 2026Stage-adaptive Token Selection for Efficient Omni-modal LLMscited by 3
  5. 2026AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compressioncited by 3
  6. 2026Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioningcited by 3
  7. 2026ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationshipscited by 3
  8. 2026OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Modelscited by 3
  9. 2026PersonalHomeBench: Evaluating Agents in Personalized Smart Homescited by 2
  10. 2026Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thoughtcited by 2
  11. 2026SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Modelscited by 2
  12. 2026OmniCap-IF: Benchmarking and Improving Instruction Following Abilities for Omni-Video Captioningcited by 2
  13. 2026AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioningcited by 2
  14. 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videoscited by 2
  15. 2026TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videoscited by 1
  16. 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioningcited by 1
  17. 2026From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMscited by 1
  18. 2026OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Modelscited by 1
  19. 2026OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMscited by 1
  20. 2026Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interactioncited by 1
  21. 2026DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learningcited by 0
  22. 2026Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videoscited by 0
  23. 2026video-SALMONN-R3: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understandingcited by 0
  24. 2026OmniKVQuant: KV Cache Quantization for Omni-LLMscited by 0
  25. 2026OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialoguecited by 0
  26. 2026OmniFysics-Captioner Technical Report: Grounding Omni-Modal Understanding in the Physical World for Better Captioningcited by 0
  27. 2026OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agentscited by 0
  28. 2026OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoningcited by 0
  29. 2026OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoningcited by 0
  30. 2026TiTok: Audio-Visual LLM for Multi-Segment Temporal Groundingcited by 0
  31. 2026SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understandingcited by 0
  32. 2026OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioningcited by 0
  33. 2026Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generationcited by 0
  34. 2026PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summarycited by 0
  35. 2026Designing Textual Interfaces To Enhance Llm Reasoning On Audio-Visual Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization
  2. 2026Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
  3. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
  4. 2024Distilling Vision-Language Models on Millions of Videos
  5. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
  6. 2023A Simple LLM Framework for Long-Range Video Question-Answering
  7. 2025TextVidBench: A Benchmark for Long Video Scene Text Understanding
  8. 2024VidCtx: Context-aware Video Question Answering with Image Models