mlsanityData to 14 Oct 2026

Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos

arXiv 2501.04001 ↗First posted 7 Jan 2025cs.CVcited by 159

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This work presents Sa2VA, the first comprehensive, unified model for dense grounded understanding of both images and videos. Unlike existing multi-modal large language models, which are often limited to specific modalities and tasks, Sa2VA supports a wide range of image and video tasks, including referring segmentation and conversation, with minimal single-stage instruction tuning. Sa2VA combines SAM-2, a foundation video segmentation model, with MLLM, advanced vision-language models, and unifies text, image, and video into a shared LLM token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2 in producing precise masks, enabling a grounded, multi-modal understanding of both static and dynamic visual content. Additionally, we introduce Ref-SAV, an auto-labeled dataset containing over 72k object expressions in complex video scenes, designed to boost model performance. We also manually validate 2k video objects in the Ref-SAV datasets to benchmark referring video object segmentation in complex environments. Experiments show that Sa2VA achieves strong performance across multiple tasks, particularly in referring video object segmentation, highlighting its potential for complex real-world applications. In addition, Sa2VA can be easily extended into various MLLMs, including Qwen-VL and Intern-VL, which can be updated with rapid process in current open-sourced MLLMs. Code and models have been provided to the community.

Key facts. Cited by 159 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 105 of its 130 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

105 of 130 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models18 Dec
  2. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos29 Sep
  3. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model28 Jun
  4. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models24 Jun
  5. 2024F-LMM: Grounding Frozen Large Multimodal Models9 Jun
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  7. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model21 Mar
  8. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentation26 Feb
  9. 2023GSVA: Generalized Segmentation via Multimodal Large Language Models15 Dec
  10. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models5 Dec
  11. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  12. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
5 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  2. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  4. 2023Contextual Object Detection with Multimodal Large Language Models29 May
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 88

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  9. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
38 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  2. 2024Qwen2 Technical Reportcited by 2,120
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  11. 2016Modeling Context in Referring Expressionscited by 1,063
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  13. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  15. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  16. 2016A Diagram Is Worth A Dozen Imagescited by 804
  17. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  18. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  19. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  20. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  21. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  22. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  23. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  24. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  25. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  26. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  27. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  28. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  29. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  30. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 255
  31. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  32. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  33. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  34. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  35. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192
  36. 2023Putting the Object Back into Video Object Segmentationcited by 188
  37. 2022Language as Queries for Referring Video Object Segmentationcited by 182
  38. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 72; 2026: 87.

159 papers.

Sort:

By area:

  1. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 139
  2. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 68
  3. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 49
  4. 2025Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learningcited by 44
  5. 2025SegEarth-R1: Geospatial Pixel Reasoning via Large Language Modelcited by 42
  6. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 41
  7. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  8. 2025OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Datacited by 37
  9. 2025BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanationcited by 33
  10. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  11. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  12. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoningcited by 28
38 more
  1. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  2. 2025Advancing Complex Video Object Segmentation via Progressive Concept Constructioncited by 26
  3. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  4. 2025EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLMcited by 18
  5. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  6. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  7. 2025DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistencycited by 15
  8. 2025OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Promptscited by 15
  9. 2025Reinforcing Video Reasoning Segmentation to Think Before It Segmentscited by 15
  10. 2025UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalistcited by 15
  11. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 15
  12. 2025Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Modelscited by 14
  13. 2025V2-SAM: Marrying SAM2 with Multi-Prompt Experts for Cross-View Object Correspondencecited by 14
  14. 2025Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Modelscited by 13
  15. 2025Object-centric Video Question Answering with Visual Grounding and Referringcited by 13
  16. 2025Region-based Cluster Discrimination for Visual Representation Learningcited by 13
  17. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D Worldcited by 12
  18. 2025Fine-grained Spatiotemporal Grounding on Egocentric Videoscited by 11
  19. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  20. 2025Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmarkcited by 11
  21. 2026SAMTok: Representing Any Mask with Two Wordscited by 11
  22. 2025Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Promptingcited by 10
  23. 2025UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretationcited by 10
  24. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  25. 2025Dense360: Dense Understanding from Omnidirectional Panoramascited by 9
  26. 2026Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentationcited by 9
  27. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Modelscited by 8
  28. 2025AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generationcited by 8
  29. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  30. 2026Venus-DeFakerOne: Unified Fake Image Detection & Localizationcited by 8
  31. 2025The 1st Solution for 7th LSVOS RVOS Track: SaSaSa2VAcited by 7
  32. 2025I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Modelscited by 7
  33. 2025InterRVOS: Interaction-aware Referring Video Object Segmentationcited by 6
  34. 2025EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphscited by 6
  35. 2025SafePLUG: Empowering Multimodal LLMs with Pixel-Level Insight and Temporal Grounding for Traffic Accident Understandingcited by 6
  36. 2026MultiAnimate: Pose-Guided Image Animation Made Extensiblecited by 6
  37. 2026Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligencecited by 6
  38. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6

50 most cited of 159 All 159 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation
  2. 2024Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models
  3. 2024ReferEverything: Towards Segmenting Everything We Can Speak of in Videos
  4. 2024CogVLM2: Visual Language Models for Image and Video Understanding
  5. 2024Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation
  6. 2025Unleashing Hierarchical Reasoning: An LLM-Driven Framework for Training-Free Referring Video Object Segmentation
  7. 2025Correspondence as Video: Test-Time Adaption on SAM2 for Reference Segmentation in the Wild
  8. 2026Event-Driven Refresh and Recurrence Memory to Reduce Stale Grounding in Referring Video Object Segmentation