mlsanityData to 7 Oct 2026

VISA: Reasoning Video Object Segmentation via Large Language Models

arXiv 2407.11325 ↗First posted 16 Jul 2024cs.CVcited by 123

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Existing Video Object Segmentation (VOS) relies on explicit user instructions, such as categories, masks, or short phrases, restricting their ability to perform complex video segmentation requiring reasoning with world knowledge. In this paper, we introduce a new task, Reasoning Video Object Segmentation (ReasonVOS). This task aims to generate a sequence of segmentation masks in response to implicit text queries that require complex reasoning abilities based on world knowledge and video contexts, which is crucial for structured environment understanding and object-centric interactions, pivotal in the development of embodied AI. To tackle ReasonVOS, we introduce VISA (Video-based large language Instructed Segmentation Assistant), to leverage the world knowledge reasoning capabilities of multi-modal LLMs while possessing the ability to segment and track objects in videos with a mask decoder. Moreover, we establish a comprehensive benchmark consisting of 35,074 instruction-mask sequence pairs from 1,042 diverse videos, which incorporates complex world knowledge reasoning into segmentation tasks for instruction-tuning and evaluation purposes of ReasonVOS models. Experiments conducted on 8 datasets demonstrate the effectiveness of VISA in tackling complex reasoning segmentation and vanilla referring segmentation in both video and image domains. The code and dataset are available at https://github.com/cilinyan/VISA.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 61 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

50 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding27 Jun
  2. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  3. 2023DetGPT: Detect What You Need via Reasoning23 May

Other references · 47

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
35 more
  1. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  5. 2023Segment Everything Everywhere All at Oncecited by 546
  6. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  7. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  8. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  9. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  10. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  11. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  12. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  13. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  14. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  15. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  16. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  17. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 232
  18. 2021Associating Objects with Transformers for Video Object Segmentationcited by 225
  19. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  20. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  21. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  22. 2021Mask2Former for Video Instance Segmentationcited by 169
  23. 2020TAO: A Large-Scale Benchmark for Tracking Any Objectcited by 146
  24. 2021End-to-End Referring Video Object Segmentation with Multimodal Transformerscited by 135
  25. 2021Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentationcited by 133
  26. 2023DVIS: Decoupled Video Instance Segmentation Frameworkcited by 77
  27. 2023OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentationcited by 62
  28. 2018Fast and Accurate Online Video Object Segmentation via Tracking Partscited by 57
  29. 2022Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentationcited by 50
  30. 2023DVIS++: Improved Decoupled Framework for Universal Video Segmentationcited by 46
  31. 2023Towards Open-Vocabulary Video Instance Segmentationcited by 45
  32. 2024DVIS-DAQ: Improving Video Segmentation via Dynamic Anchor Queriescited by 22
  33. 2024TrackGPT -- A generative pre-trained transformer for cross-domain entity trajectory forecastingcited by 13
  34. 2021Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challengecited by 6
  35. 20231st Place Solution for the 5th LSVOS Challenge: Video Instance Segmentationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 66; 2026: 45.

123 papers.

Sort:

By area:

  1. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  2. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  3. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
  4. 2025Efficient Reasoning with Hidden Thinkingcited by 72
  5. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  6. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 59
  7. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 52
  8. 2024SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentationcited by 43
  9. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 41
  10. 2025Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaborationcited by 38
  11. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34
  12. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
38 more
  1. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Modelscited by 31
  2. 2025The Devil is in Temporal Token: High Quality Video Reasoning Segmentationcited by 26
  3. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 26
  4. 2025Advancing Complex Video Object Segmentation via Progressive Concept Constructioncited by 24
  5. 2024ViLLa: Video Reasoning Segmentation with Large Language Modelcited by 23
  6. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  7. 2025Multimodal Referring Segmentation: A Surveycited by 22
  8. 2026Large Language Model Reasoning Failurescited by 20
  9. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 19
  10. 2024Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Levelcited by 18
  11. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  12. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  13. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  14. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  15. 2025VideoMolmo: Spatio-Temporal Grounding Meets Pointingcited by 14
  16. 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
  17. 2025A Large-Scale Referring Remote Sensing Image Segmentation Dataset and Benchmarkcited by 12
  18. 2025Object-centric Video Question Answering with Visual Grounding and Referringcited by 12
  19. 2026Video Generation Models are General-Purpose Vision Learnerscited by 12
  20. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentationcited by 11
  21. 2025Online Reasoning Video Segmentation with Just-in-Time Digital Twinscited by 11
  22. 2025Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learningcited by 11
  23. 2025Fine-grained Spatiotemporal Grounding on Egocentric Videoscited by 11
  24. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  25. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 11
  26. 2024ReferEverything: Towards Segmenting Everything We Can Speak of in Videoscited by 10
  27. 2025Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentationcited by 10
  28. 2025Deforming Videos to Masks: Flow Matching for Referring Video Segmentationcited by 10
  29. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  30. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
  31. 2025RVTBench: A Benchmark for Visual Reasoning Taskscited by 9
  32. 2024ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentationcited by 8
  33. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  34. 2025PromptLNet: Region-Adaptive Aesthetic Enhancement via Prompt Guidance in Low-Light Enhancement Netcited by 8
  35. 2025CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videoscited by 8
  36. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Modelscited by 8
  37. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 8
  38. 2025MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentationcited by 7

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos
  2. 2025Reinforcing Video Reasoning Segmentation to Think Before It Segments
  3. 2026Training-Free Spatio-temporal Decoupled Reasoning Video Segmentation with Adaptive Object Memory
  4. 2026AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentation
  5. 2025Reasoning Segmentation for Images and Videos: A Survey
  6. 2026VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation
  7. 2025Fast Reasoning Segmentation for Images and Videos
  8. 2026Report of the 8th LSVOS Challenge: Complex and Multimodal Video Object Segmentation