mlsanityData to 7 Oct 2026

Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time

arXiv 2407.01851 ↗First posted 1 Jul 2024cs.CV cs.AI cs.LGcited by 26

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Leveraging Large Language Models' remarkable proficiency in text-based tasks, recent works on Multi-modal LLMs (MLLMs) extend them to other modalities like vision and audio. However, the progress in these directions has been mostly focused on tasks that only require a coarse-grained understanding of the audio-visual semantics. We present Meerkat, an audio-visual LLM equipped with a fine-grained understanding of image and audio both spatially and temporally. With a new modality alignment module based on optimal transport and a cross-attention module that enforces audio-visual consistency, Meerkat can tackle challenging tasks such as audio referred image grounding, image guided audio temporal localization, and audio-visual fact-checking. Moreover, we carefully curate a large dataset AVFIT that comprises 3M instruction tuning samples collected from open-source datasets, and introduce MeerkatBench that unifies five challenging audio-visual tasks. We achieve state-of-the-art performance on all these downstream tasks with a relative improvement of up to 37.12%.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 98 of its 120 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

98 of 120 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Audio-Visual LLM for Video Understanding11 Dec
  2. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 95

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  13. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  14. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  15. 2021Grounded Language-Image Pre-trainingcited by 1,147
  16. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  17. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  18. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  20. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  21. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  22. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  23. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  24. 2022Galactica: A Large Language Model for Sciencecited by 736
  25. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  26. 2023Instruction Tuning with GPT-4cited by 676
  27. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  28. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  29. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  30. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  31. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  32. 2017Look, Listen and Learncited by 452
  33. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  34. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  35. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  36. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  37. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  38. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 9.

26 papers.

Sort:

By area:

  1. 2024EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialoguescited by 81
  2. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  3. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 26
  4. 2025AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMscited by 19
  5. 2025Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 16
  6. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 15
  7. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 11
  8. 2025Aligned Better, Listen Better for Audio-Visual Large Language Modelscited by 10
  9. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  10. 2025Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Modelscited by 7
  11. 2025DAVE: Diagnostic benchmark for Audio Visual Evaluationcited by 5
  12. 2026FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inferencecited by 5
14 more
  1. 2025EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perceptioncited by 4
  2. 2025AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understandingcited by 4
  3. 2026SonoWorld: From One Image to a 3D Audio-Visual Scenecited by 3
  4. 2026Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMscited by 3
  5. 2025Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Modelcited by 2
  6. 2025Agentic Design Review Systemcited by 2
  7. 2026Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videoscited by 2
  8. 2024TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorizationcited by 1
  9. 2025TEn-CATG:Text-Enriched Audio-Visual Video Parsing with Multi-Scale Category-Aware Temporal Graphcited by 1
  10. 2026Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMscited by 1
  11. 2026Crab+: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperationcited by 0
  12. 2026AV-Unified: A Unified Framework for Audio-visual Scene Understandingcited by 0
  13. 2026Through the PRISM: Principle-Aware, Interpretable, and Multi-Scale Evaluation of Visual Designscited by 0
  14. 2026Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Audio-visual training for improved grounding in video-text LLMs
  2. 2026Do Audio-Visual Large Language Models Really See and Hear?
  3. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMs
  4. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities
  5. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision
  6. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  7. 2026AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models
  8. 2024Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding