mlsanityData to 14 Oct 2026

GSVA: Generalized Segmentation via Multimodal Large Language Models

arXiv 2312.10103 ↗First posted 15 Dec 2023cs.CVcited by 154

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Generalized Referring Expression Segmentation (GRES) extends the scope of classic RES to refer to multiple objects in one expression or identify the empty targets absent in the image. GRES poses challenges in modeling the complex spatial relationships of the instances in the image and identifying non-existing referents. Multimodal Large Language Models (MLLMs) have recently shown tremendous progress in these complicated vision-language tasks. Connecting Large Language Models (LLMs) and vision models, MLLMs are proficient in understanding contexts with visual inputs. Among them, LISA, as a representative, adopts a special [SEG] token to prompt a segmentation mask decoder, e.g., SAM, to enable MLLMs in the RES task. However, existing solutions to GRES remain unsatisfactory since current segmentation MLLMs cannot correctly handle the cases where users might reference multiple subjects in a singular prompt or provide descriptions incongruent with any image target. In this paper, we propose Generalized Segmentation Vision Assistant (GSVA) to address this gap. Specifically, GSVA reuses the [SEG] token to prompt the segmentation model towards supporting multiple mask references simultaneously and innovatively learns to generate a [REJ] token to reject the null targets explicitly. Experiments validate GSVA's efficacy in resolving the GRES issue, marking a notable enhancement and setting a new record on the GRES benchmark gRefCOCO dataset. GSVA also proves effective across various classic referring segmentation and comprehension tasks.

Key facts. Cited by 154 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 83 of its 104 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

83 of 104 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premises13 Dec
  2. 2023NExT-Chat: An LMM for Chat, Detection and Segmentation8 Nov
  3. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  4. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 77

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Segment Anythingcited by 8,234
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  5. 2016Modeling Context in Referring Expressionscited by 1,063
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  8. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  10. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  12. 2023Segment Everything Everywhere All at Oncecited by 548
  13. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 522
  14. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  15. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  16. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  17. 2021Dynamic Neural Networks: A Surveycited by 408
  18. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  19. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  20. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  21. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  22. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  23. 2022Vision Transformer with Deformable Attentioncited by 295
  24. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  25. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  26. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  27. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  28. 2016Segmentation from Natural Language Expressionscited by 257
  29. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  30. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  31. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 232
  32. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  33. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 172
  34. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 148
  35. 2023PACO: Parts and Attributes of Common Objectscited by 146
  36. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 144
  37. 2020Referring Image Segmentation via Cross-Modal Progressive Comprehensioncited by 131
  38. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 24; 2025: 77; 2026: 53.

154 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  2. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  3. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  4. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 108
  5. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 101
  6. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  7. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 69
  8. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 65
  9. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 61
  10. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 59
  11. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  12. 2024Toward Visual Grounding: A Surveycited by 52
38 more
  1. 2024Bridging the Divide: Reconsidering Softmax and Linear Attentioncited by 48
  2. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  3. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  4. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  5. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  6. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Modelscited by 31
  7. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 30
  8. 2025MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentationcited by 28
  9. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  10. 2025The Devil is in Temporal Token: High Quality Video Reasoning Segmentationcited by 26
  11. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediatorscited by 24
  12. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  13. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  14. 2025Multimodal Referring Segmentation: A Surveycited by 22
  15. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 20
  16. 2025ViT3: Unlocking Test-Time Training in Visioncited by 18
  17. 2025SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenescited by 17
  18. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 15
  19. 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
  20. 2024High-Quality Entity Segmentation and Groundingcited by 13
  21. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  22. 2025LISAT: Language-Instructed Segmentation Assistant for Satellite Imagerycited by 13
  23. 2025DeRIS: Decoupling Perception and Cognition for Enhanced Referring Image Segmentation through Loopback Synergycited by 13
  24. 2025Region-based Cluster Discrimination for Visual Representation Learningcited by 13
  25. 2025RationalVLA: A Rational Vision-Language-Action Model with Dual Systemcited by 12
  26. 2025GroundingSuite: Measuring Complex Multi-Granular Pixel Groundingcited by 11
  27. 2025EIAD: Explainable Industrial Anomaly Detection Via Multi-Modal Large Language Modelscited by 11
  28. 2025Online Reasoning Video Segmentation with Just-in-Time Digital Twinscited by 11
  29. 2024CoHD: A Counting-Aware Hierarchical Decoding Framework for Generalized Referring Expression Segmentationcited by 10
  30. 2024Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehensioncited by 10
  31. 2025Where, What, Why: Towards Explainable Driver Attention Predictioncited by 10
  32. 2025PropVG: End-to-End Proposal-Driven Visual Grounding with Multi-Granularity Discriminationcited by 10
  33. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  34. 2025Operating Room Workflow Analysis via Reasoning Segmentation over Digital Twinscited by 9
  35. 2025CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videoscited by 9
  36. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 9
  37. 2024Bring Adaptive Binding Prototypes to Generalized Referring Expression Segmentationcited by 8
  38. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8

50 most cited of 154 All 154 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder
  2. 2025Deformable Attentive Visual Enhancement for Referring Segmentation Using Vision-Language Model
  3. 2026GenSeg-R1: RL-Driven Vision-Language Grounding for Fine-Grained Referring Segmentation
  4. 2026ResAgent: Entropy-based Prior Point Discovery and Visual Reasoning for Referring Expression Segmentation
  5. 2025X-SAM: From Segment Anything to Any Segmentation
  6. 2025GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation
  7. 2026X2SAM: Any Segmentation in Images and Videos
  8. 2023A Simple Recipe for Language-guided Domain Generalized Segmentation