mlsanityData to 14 Oct 2026

VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos

arXiv 2411.04923 ↗First posted 7 Nov 2024cs.CVcited by 51

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Fine-grained alignment between videos and text is challenging due to complex spatial and temporal dynamics in videos. Existing video-based Large Multimodal Models (LMMs) handle basic conversations but struggle with precise pixel-level grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed for fine-grained pixel-level grounding in videos based on user-provided textual inputs. Our design seamlessly connects three key components: a Large Language Model, a dual vision encoder that emphasizes both spatial and temporal details, and a spatio-temporal decoder for accurate mask generation. This connection is facilitated via tunable V-L and L-V adapters that enable close Vision-Language (VL) alignment. The architecture is trained to synchronize both spatial and temporal elements of video content with textual instructions. To enable fine-grained grounding, we curate a multimodal dataset featuring detailed visually-grounded conversations using a semiautomatic annotation pipeline, resulting in a diverse set of 38k video-QA triplets along with 83k objects and 671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded Conversation Generation, Visual Grounding, and Referring Video Segmentation. Experimental results show that our model consistently outperforms existing approaches across all three tasks.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 60 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

48 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding13 Jun
  2. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Models22 Nov
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  4. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Segment Anythingcited by 8,234
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
32 more
  1. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  8. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  10. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  11. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  12. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  13. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  14. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  15. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  16. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  17. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  18. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  19. 2018Video Object Segmentation with Language Referring Expressionscited by 162
  20. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 146
  21. 2023PACO: Parts and Attributes of Common Objectscited by 146
  22. 2020Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentencescited by 118
  23. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  24. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  25. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videoscited by 100
  26. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  27. 2020Human-centric Spatio-Temporal Video Grounding With Visual Transformerscited by 85
  28. 2018Grounded Video Descriptioncited by 78
  29. 2022BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Videocited by 66
  30. 2023Tracking with Human-Intent Reasoningcited by 33
  31. 2022Video Mask Transfiner for High-Quality Video Instance Segmentationcited by 28
  32. 2023BIVDiff: A Training-Free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Modelscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 22; 2026: 28.

51 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  2. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 68
  3. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  4. 2025GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentationcited by 34
  5. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoningcited by 28
  6. 2025FetalCLIP: A Visual-Language Foundation Model for Fetal Ultrasound Image Analysiscited by 24
  7. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  8. 2025Multimodal Referring Segmentation: A Surveycited by 22
  9. 2025VideoMolmo: Spatio-Temporal Grounding Meets Pointingcited by 14
  10. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  11. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D Worldcited by 12
  12. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
38 more
  1. 2025iMOVE: Instance-Motion-Aware Video Understandingcited by 10
  2. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Modelscited by 8
  3. 2025Steering Multimodal Large Language Models Decoding for Context-Aware Safetycited by 7
  4. 2025InterRVOS: Interaction-aware Referring Video Object Segmentationcited by 6
  5. 2025SafePLUG: Empowering Multimodal LLMs with Pixel-Level Insight and Temporal Grounding for Traffic Accident Understandingcited by 6
  6. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  7. 2025Large-scale Pre-training for Grounded Video Caption Generationcited by 5
  8. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4
  9. 2026Instance-Aligned Captions for Explainable Video Anomaly Detectioncited by 4
  10. 2024Referring Video Object Segmentation via Language-aligned Track Selectioncited by 3
  11. 2026Enhancing Spatial Understanding in Image Generation via Reward Modelingcited by 3
  12. 2025UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Modelscited by 2
  13. 2026SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMscited by 2
  14. 2026Beyond a Single Frame: Multi-Frame Spatially Grounded Reasoning Across Volumetric MRIcited by 2
  15. 2026SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Predictioncited by 2
  16. 2026VideoKR: Towards Knowledge- and Reasoning-Intensive Video Understandingcited by 2
  17. 2026Evidence-Backed Video Question Answeringcited by 2
  18. 2026Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understandingcited by 2
  19. 2025PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?cited by 1
  20. 2025SVAC: Scaling Is All You Need For Referring Video Object Segmentationcited by 1
  21. 2026VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentationcited by 1
  22. 2026X2SAM: Any Segmentation in Images and Videoscited by 1
  23. 2026Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarificationcited by 1
  24. 2026An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentationcited by 1
  25. 2026TAU-Bench: From Anomaly Instance Tracking to Fine-Grained Video Anomaly Understandingcited by 1
  26. 2025Planner-Refiner: Dynamic Space-Time Refinement for Vision-Language Alignment in Videoscited by 0
  27. 2026Scene Graph-guided SegCaptioning Transformer with Fine-grained Alignment for Controllable Video Segmentation and Captioningcited by 0
  28. 2026Online Reasoning Video Object Segmentationcited by 0
  29. 2026One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognitioncited by 0
  30. 2026Reward-Decomposed Reinforcement Learning for Immersive Video Role-Playingcited by 0
  31. 2026CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Modelscited by 0
  32. 2026GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoningcited by 0
  33. 2026VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentationcited by 0
  34. 2026Boosting Text-Driven Video Segmentation via Geometry-Aware Distillationcited by 0
  35. 2026FeVOS: Foresight Expression Video Object Segmentationcited by 0
  36. 2026Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verificationcited by 0
  37. 2026PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videoscited by 0
  38. 2026Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Groundingcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervision
  2. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Grounding
  3. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models
  4. 2026GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
  5. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Model
  6. 2024B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokens
  7. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  8. 2025SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capability