mlsanityData to 14 Oct 2026

Audio-Visual LLM for Video Understanding

arXiv 2312.06720 ↗First posted 11 Dec 2023cs.CVcited by 77

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

This paper presents Audio-Visual LLM, a Multimodal Large Language Model that takes both visual and auditory inputs for holistic video understanding. A key design is the modality-augmented training, which involves the integration of modality-specific tokens engineered to activate the appropriate visual and/or auditory encoder selectively. This mechanism is pivotal in enabling end-to-end joint training with video data at different modalities, including visual-only, audio-only, and audio-visual formats. Moreover, we introduce a high-quality video instruction dataset, derived from GPT-4. This dataset allows Audio-Visual LLM to adeptly process a variety of task-oriented video instructions, ranging from multi-turn conversations and audio-visual narratives to complex reasoning tasks. Extensive experiments demonstrate that Audio-Visual LLM impressively achieves strong zero-shot results across a range of video understanding tasks. For example, Audio-Visual LLM achieves an accuracy of 53.7% on MSRVTT-QA, outperforming non-LLM-based InterVideo by 6.6% and LLM-based Valley by 4.4%, respectively. Additionally, our Audio-Visual LLM also achieves competitive performance on audio tasks (e.g., AudioCaps).

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 84 of its 102 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

84 of 102 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  2. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models9 Oct
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 81

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  9. 2021ViViT: A Video Vision Transformercited by 1,620
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  13. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  14. 2023PaLM 2 Technical Reportcited by 1,162
  15. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  16. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  17. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  18. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  19. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  20. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  21. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  22. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  23. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  24. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  25. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  26. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  27. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  28. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  29. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  30. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  31. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  32. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  33. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  34. 2019Clotho: An Audio Captioning Datasetcited by 358
  35. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  36. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  37. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  38. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 21; 2025: 34; 2026: 21.

77 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  3. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  4. 2023Video Understanding with Large Language Models: A Surveycited by 206
  5. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  6. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 102
  7. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  8. 2024LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videoscited by 51
  9. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  10. 2024V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuningcited by 46
  11. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 45
  12. 2025ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shortscited by 41
38 more
  1. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  2. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 31
  3. 2025EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learningcited by 29
  4. 2024Taming Data and Transformers for Audio Generationcited by 28
  5. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  6. 2025SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understandingcited by 25
  7. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  8. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  9. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 24
  10. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  11. 2025Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skillscited by 22
  12. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 22
  13. 2025AdaTooler-V: Adaptive Tool-Use for Images and Videoscited by 21
  14. 2024Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understandingcited by 18
  15. 2024Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMscited by 17
  16. 2025JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluationcited by 17
  17. 2025JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generationcited by 15
  18. 2024Do Language Models Understand Time?cited by 14
  19. 2025Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thoughtcited by 14
  20. 2025SAIL-VL2 Technical Reportcited by 14
  21. 2024Patch-level Sounding Object Tracking for Audio-Visual Question Answeringcited by 13
  22. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  23. 2026DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Modelscited by 13
  24. 2024VideoOrion: Tokenizing Object Dynamics in Videoscited by 12
  25. 2025Aligned Better, Listen Better for Audio-Visual Large Language Modelscited by 10
  26. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  27. 2025IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioningcited by 7
  28. 2025UniComp: Rethinking Video Compression Through Informational Uniquenesscited by 7
  29. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  30. 2025LASER: Lip Landmark Assisted Speaker Detection for Robustnesscited by 6
  31. 2025MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modalitycited by 6
  32. 2025"Before, I Asked My Mom, Now I Ask ChatGPT": Visual Privacy Management with Generative AI for Blind and Low-Vision Peoplecited by 6
  33. 2025EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMscited by 6
  34. 2026DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compressioncited by 6
  35. 2024Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognitioncited by 5
  36. 2026MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Freecited by 5
  37. 2026OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answeringcited by 5
  38. 2024Reframe Anything: LLM Agent for Open World Video Reframingcited by 4

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Audio-visual training for improved grounding in video-text LLMs
  2. 2025Audio-centric Video Understanding Benchmark without Text Shortcut
  3. 2024From Image to Video, what do we need in multimodal LLMs?
  4. 2025Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy
  5. 2026Do Audio-Visual Large Language Models Really See and Hear?
  6. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding
  7. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens
  8. 2025Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models