mlsanityData to 7 Oct 2026

CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios

arXiv 2403.04640 ↗First posted 7 Mar 2024cs.CVcited by 53

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

This paper focuses on the challenge of answering questions in scenarios that are composed of rich and complex dynamic audio-visual components. Although existing Multimodal Large Language Models (MLLMs) can respond to audio-visual content, these responses are sometimes ambiguous and fail to describe specific audio-visual events. To overcome this limitation, we introduce the CAT, which enhances MLLM in three ways: 1) besides straightforwardly bridging audio and video, we design a clue aggregator that aggregates question-related clues in dynamic audio-visual scenarios to enrich the detailed knowledge required for large language models. 2) CAT is trained on a mixed multimodal dataset, allowing direct application in audio-visual scenarios. Notably, we collect an audio-visual joint instruction dataset named AVinstruct, to further enhance the capacity of CAT to model cross-semantic correlations. 3) we propose AI-assisted ambiguity-aware direct preference optimization, a strategy specialized in retraining the model to favor the non-ambiguity response and improve the ability to localize specific audio-visual objects. Extensive experimental results demonstrate that CAT outperforms existing methods on multimodal tasks, especially in Audio-Visual Question Answering (AVQA) tasks. The codes and the collected instructions are released at https://github.com/rikeilong/Bay-CAT.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 68 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

59 of 68 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  9. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  10. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  11. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  12. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  13. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  14. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  15. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  16. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  17. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  18. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  19. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  20. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  21. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  22. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  23. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  24. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  25. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  26. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  27. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  28. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  29. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  30. 2019Audio-Visual Scene-Aware Dialogcited by 122
  31. 2021Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videoscited by 86
  32. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  33. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  34. 2023Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokenscited by 70
  35. 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
  36. 2019A Simple Baseline for Audio-Visual Scene-Aware Dialogcited by 52
  37. 2019Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systemscited by 52
  38. 2019Temporal Reasoning via Audio Question Answeringcited by 38

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 25; 2026: 19.

53 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  4. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  5. 2024FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mambacited by 81
  6. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  7. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  8. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 50
  9. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agentcited by 35
  10. 2025Audio-centric Video Understanding Benchmark without Text Shortcutcited by 33
  11. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  12. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 26
38 more
  1. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  2. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  3. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  4. 2026DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Modelscited by 12
  5. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  6. 2024PlanLLM: Video Procedure Planning with Refinable Large Language Modelscited by 10
  7. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  8. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 9
  9. 2025VideoExpert: Augmented LLM for Temporal-Sensitive Video Understandingcited by 8
  10. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  11. 2025Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Modelscited by 7
  12. 2025DAVE: Diagnostic benchmark for Audio Visual Evaluationcited by 5
  13. 2026FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inferencecited by 5
  14. 2026Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Modelscited by 5
  15. 2025PAVE: Patching and Adapting Video Large Language Modelscited by 4
  16. 2025Music Audio-Visual Question Answering Requires Specialized Multimodal Designscited by 4
  17. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMscited by 4
  18. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustnesscited by 3
  19. 2026Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMscited by 3
  20. 2026DRS-GUI: Dynamic Region Search for Training-Free GUI Groundingcited by 3
  21. 2025COP-GEN-Beta: Unified Generative Modelling of COPernicus Imagery Thumbnailscited by 2
  22. 2025VAT-KG: Knowledge-Intensive Multimodal Knowledge Graph Dataset for Retrieval-Augmented Generationcited by 2
  23. 2025Agentic Design Review Systemcited by 2
  24. 2025IAD-GPT: Advancing Visual Knowledge in Multimodal Large Language Model for Industrial Anomaly Detectioncited by 2
  25. 2025AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answeringcited by 2
  26. 2026MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Modelscited by 2
  27. 2025Multimodal Long Video Modeling Based on Temporal Dynamic Contextcited by 1
  28. 2025LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMscited by 1
  29. 2026MAVIS: Multi-Agent Video Retrieval via Structured Video Understandingcited by 1
  30. 2026From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMscited by 1
  31. 2026MVEB: Massive Video Embedding Benchmarkcited by 1
  32. 2026CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scalescited by 1
  33. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  34. 2026Omni-Streaming Thinkingcited by 1
  35. 2026Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMscited by 1
  36. 2025An Enhanced Large Language Model For Cross Modal Query Understanding System Using DL-KeyBERT Based CAZSSCL-MPGPTcited by 0
  37. 2026Acoustic Field Video for Multimodal Scene Understandingcited by 0
  38. 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
  2. 2025FortisAVQA and MAVEN: a Benchmark Dataset and Debiasing Framework for Robust Multimodal Reasoning
  3. 2025Chain of Questions: Guiding Multimodal Curiosity in Language Models
  4. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
  5. 2026SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
  6. 2025Question-Aware Gaussian Experts for Audio-Visual Question Answering
  7. 2024Cross-modal Information Flow in Multimodal Large Language Models
  8. 2024SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization