mlsanityData to 7 Oct 2026

InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models

arXiv 2412.14006 ↗First posted 18 Dec 2024cs.CVcited by 31

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Boosted by Multi-modal Large Language Models (MLLMs), text-guided universal segmentation models for the image and video domains have made rapid progress recently. However, these methods are often developed separately for specific domains, overlooking the similarities in task settings and solutions across these two areas. In this paper, we define the union of referring segmentation and reasoning segmentation at both the image and video levels as Instructed Visual Segmentation (IVS). Correspondingly, we propose InstructSeg, an end-to-end segmentation pipeline equipped with MLLMs for IVS. Specifically, we employ an object-aware video perceiver to extract temporal and object information from reference frames, facilitating comprehensive video understanding. Additionally, we introduce vision-guided multi-granularity text fusion to better integrate global and detailed text information with fine-grained visual guidance. By leveraging multi-task and end-to-end training, InstructSeg demonstrates superior performance across diverse image and video segmentation tasks, surpassing both segmentation specialists and MLLM-based methods with a single model. Our code is available at https://github.com/congvvc/InstructSeg.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 57 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

20 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model21 Mar
  2. 2023GSVA: Generalized Segmentation via Multimodal Large Language Models15 Dec
  3. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 16

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  6. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  7. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  8. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  9. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  10. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 116
  11. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  12. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
4 more
  1. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  2. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  3. 2024Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Modelscited by 24
  4. 2024TrackGPT -- A generative pre-trained transformer for cross-domain entity trajectory forecastingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 16.

31 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  2. 2025SegEarth-R1: Geospatial Pixel Reasoning via Large Language Modelcited by 40
  3. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoningcited by 27
  4. 2025Multimodal Referring Segmentation: A Surveycited by 22
  5. 2025LENS: Learning to Segment Anything with Unified Reinforced Reasoningcited by 18
  6. 2025Reinforcing Video Reasoning Segmentation to Think Before It Segmentscited by 15
  7. 2025Reasoning Segmentation for Images and Videos: A Surveycited by 12
  8. 2025GroundingSuite: Measuring Complex Multi-Granular Pixel Groundingcited by 10
  9. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  10. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
  11. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 8
  12. 2025Counterfactual Segmentation Reasoning: Diagnosing and Mitigating Pixel-Grounding Hallucinationcited by 7
19 more
  1. 2026Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentationcited by 7
  2. 2026InstructSAM: Segment Any Instance with Any Instructionscited by 6
  3. 2025Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentationcited by 5
  4. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4
  5. 2025UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Modelscited by 2
  6. 2026SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Predictioncited by 2
  7. 2025EVTP-IVS: Effective Visual Token Pruning For Unifying Instruction Visual Segmentation In Multi-Modal Large Language Modelscited by 1
  8. 2026OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transportcited by 1
  9. 2026VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentationcited by 1
  10. 2026Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPOcited by 1
  11. 2026Show Me When and Where: Towards Referring Video Object Segmentation in the Wildcited by 0
  12. 2026Weakly-Supervised Referring Video Object Segmentation through Text Supervisioncited by 0
  13. 2026Learning to Label: A Reinforced Self-Evolving Framework for Semi-supervised Referring Expression Segmentationcited by 0
  14. 2026VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentationcited by 0
  15. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  16. 2026PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videoscited by 0
  17. 2026Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentationcited by 0
  18. 2026PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selectioncited by 0
  19. 2026CoEvolve: Construct-to-Edit Visual Grounding with Bidirectional State Refinementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos
  2. 2026X2SAM: Any Segmentation in Images and Videos
  3. 2026Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation
  4. 2026Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
  5. 2025X-SAM: From Segment Anything to Any Segmentation
  6. 2025Unleashing Hierarchical Reasoning: An LLM-Driven Framework for Training-Free Referring Video Object Segmentation
  7. 2026Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning
  8. 2023IFSeg: Image-free Semantic Segmentation via Vision-Language Model