mlsanityData to 7 Oct 2026

video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models

arXiv 2406.15704 ↗First posted 22 Jun 2024cs.CVcited by 101

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Speech understanding as an element of the more generic video understanding using audio-visual large language models (av-LLMs) is a crucial yet understudied aspect. This paper proposes video-SALMONN, a single end-to-end av-LLM for video processing, which can understand not only visual frame sequences, audio events and music, but speech as well. To obtain fine-grained temporal information required by speech understanding, while keeping efficient for other video elements, this paper proposes a novel multi-resolution causal Q-Former (MRC Q-Former) structure to connect pre-trained audio-visual encoders and the backbone large language model. Moreover, dedicated training approaches including the diversity loss and the unpaired audio-visual mixed training scheme are proposed to avoid frames or modality dominance. On the introduced speech-audio-visual evaluation benchmark, video-SALMONN achieves more than 25% absolute accuracy improvements on the video-QA task and over 30% absolute accuracy improvements on audio-visual QA tasks with human speech. In addition, video-SALMONN demonstrates remarkable video comprehension and reasoning abilities on tasks that are unprecedented by other av-LLMs. Our training code and model checkpoints are available at texttturlhttps://github.com/bytedance/SALMONN/.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 22 Jun 2024.

Builds on

36 of 50 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
24 more
  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  5. 2023Instruction Tuning with GPT-4cited by 676
  6. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  7. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  8. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  9. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  10. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  11. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  12. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  13. 2022Learning Video Representations from Large Language Modelscited by 204
  14. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  15. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  16. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  17. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  18. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 112
  19. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  20. 2023Audio-Visual LLM for Video Understandingcited by 76
  21. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  22. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  23. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  24. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 54; 2026: 38.

101 papers.

Sort:

By area:

  1. 2025Qwen2.5-Omni Technical Reportcited by 704
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  4. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  5. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  6. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  7. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  8. 2024LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Modelscited by 51
  9. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  10. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  11. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  12. 2024AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?cited by 46
38 more
  1. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  2. 2025ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shortscited by 36
  3. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  4. 2025Introducing Visual Perception Token into Multimodal Large Language Modelcited by 26
  5. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 26
  6. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  7. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 25
  8. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  9. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  10. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  11. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  12. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 19
  13. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  14. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  15. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  16. 2025SATORI-R1: Incentivizing Multimodal Reasoning through Explicit Visual Anchoringcited by 17
  17. 2025Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 16
  18. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 15
  19. 2025HarmonySet: A Comprehensive Dataset for Understanding Video-Music Semantic Alignment and Temporal Synchronizationcited by 14
  20. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 14
  21. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 13
  22. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  23. 2025Survey on AI-Generated Media Detection: From Non-MLLM to MLLMcited by 12
  24. 2025JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generationcited by 12
  25. 2026DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Modelscited by 12
  26. 2025Evaluating Robustness of Large Audio Language Models to Audio Injection: An Empirical Studycited by 11
  27. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  28. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  29. 2025Aligned Better, Listen Better for Audio-Visual Large Language Modelscited by 10
  30. 2025Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentationcited by 10
  31. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  32. 2026OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attentioncited by 9
  33. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusioncited by 9
  34. 2026Do Audio-Visual Large Language Models Really See and Hear?cited by 9
  35. 2024Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimizationcited by 8
  36. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  37. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  38. 2026TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captionscited by 8

50 most cited of 101 All 101 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLM
  2. 2025Audio-centric Video Understanding Benchmark without Text Shortcut
  3. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
  4. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learners
  5. 2024Audio-visual training for improved grounding in video-text LLMs
  6. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  7. 2024Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding
  8. 2025Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy