mlsanityData to 7 Oct 2026

PG-Video-LLaVA: Pixel Grounding Large Video-Language Models

arXiv 2311.13435 ↗First posted 22 Nov 2023cs.CV cs.AIcited by 50

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Extending image-based Large Multimodal Models (LMMs) to videos is challenging due to the inherent complexity of video data. The recent approaches extending image-based LMMs to videos either lack the grounding capabilities (e.g., VideoChat, Video-ChatGPT, Video-LLaMA) or do not utilize the audio-signals for better video understanding (e.g., Video-ChatGPT). Addressing these gaps, we propose PG-Video-LLaVA, the first LMM with pixel-level grounding capability, integrating audio cues by transcribing them into text to enrich video-context understanding. Our framework uses an off-the-shelf tracker and a novel grounding module, enabling it to spatially localize objects in videos following user instructions. We evaluate PG-Video-LLaVA using video-based generative and question-answering benchmarks and introduce new benchmarks specifically designed to measure prompt-based object grounding performance in videos. Further, we propose the use of Vicuna over GPT-3.5, as utilized in Video-ChatGPT, for video-based conversation benchmarking, ensuring reproducibility of results which is a concern with the proprietary nature of GPT-3.5. Our framework builds on SoTA image-based LLaVA model and extends its advantages to the video domain, delivering promising gains on video-based conversation and grounding tasks. Project Page: https://github.com/mbzuai-oryx/Video-LLaVA

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 51 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

35 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
22 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  2. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  3. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  4. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  5. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  6. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  9. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  10. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
  11. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  12. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  13. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  14. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  15. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  16. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  17. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  18. 2020Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentencescited by 116
  19. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  20. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  21. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  22. 2020Human-centric Spatio-Temporal Video Grounding With Visual Transformerscited by 83

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 22; 2025: 16; 2026: 11.

50 papers.

Sort:

By area:

  1. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  2. 2023Video Understanding with Large Language Models: A Surveycited by 203
  3. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  4. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  5. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  6. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 99
  7. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  8. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  9. 2024ChatVTG: Video Temporal Grounding via Chat with Video Dialogue Large Language Modelscited by 54
  10. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  11. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  12. 2024Elysium: Exploring Object-level Perception in Videos via MLLMcited by 48
38 more
  1. 2024Foundation Models for Video Understanding: A Surveycited by 46
  2. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 39
  3. 2025GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentationcited by 34
  4. 2024How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMscited by 30
  5. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  6. 2024Towards Open-Vocabulary Audio-Visual Event Localizationcited by 26
  7. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  8. 2024Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Levelcited by 18
  9. 2025Is Your Video Language Model a Reliable Judge?cited by 14
  10. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  11. 2024HOI-Ref: Hand-Object Interaction Referral in Egocentric Visioncited by 9
  12. 2024Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selectioncited by 9
  13. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  14. 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Groundingcited by 9
  15. 2025CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videoscited by 8
  16. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  17. 2024Unveiling the Invisible: Captioning Videos with Metaphorscited by 7
  18. 2024PiTe: Pixel-Temporal Alignment for Large Video-Language Modelcited by 7
  19. 2025Think Before You Segment: High-Quality Reasoning Segmentation with GPT Chain of Thoughtscited by 7
  20. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  21. 2025Large-scale Pre-training for Grounded Video Caption Generationcited by 5
  22. 2025VEGAS: Towards Visually Explainable and Grounded Artificial Social Intelligencecited by 4
  23. 2025HuMoCon: Concept Discovery for Human Motion Understandingcited by 4
  24. 2026Instance-Aligned Captions for Explainable Video Anomaly Detectioncited by 4
  25. 2025Agentic Design Review Systemcited by 2
  26. 2025COLT: Enhancing Video Large Language Models with Continual Tool Usagecited by 2
  27. 2026Towards Sparse Video Understanding and Reasoningcited by 2
  28. 2025Enhancing the Learning Experience: Using Vision-Language Models to Generate Questions for Educational Videoscited by 1
  29. 2026DeepSVU: Towards In-depth Security-oriented Video Understanding via Unified Physical-world Regularized MoEcited by 1
  30. 2026X2SAM: Any Segmentation in Images and Videoscited by 1
  31. 2024Realizing Video Summarization from the Path of Language-based Semantic Understandingcited by 0
  32. 2026RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understandingcited by 0
  33. 2026Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Surveycited by 0
  34. 2026One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognitioncited by 0
  35. 2026Through the PRISM: Principle-Aware, Interpretable, and Multi-Scale Evaluation of Visual Designscited by 0
  36. 2026VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentationcited by 0
  37. 2026StateTrace: An Object-Centric Framework for Hidden-State Spatiotemporal Reasoning in Long Videoscited by 0
  38. 2026PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videoscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
  2. 2026GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
  3. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervision
  4. 2023GLaMM: Pixel Grounding Large Multimodal Model
  5. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models
  6. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMs
  7. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Model
  8. 2024Audio-visual training for improved grounding in video-text LLMs