# GSVA: Generalized Segmentation via Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2312.10103 · HTML: https://mlsanity.com/paper/2312.10103/

## Key facts

- Cited by 154 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 83 of its 104 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 83)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](https://mlsanity.com/paper/2305.11175/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)

## Built on by (most cited first, 30 of 154)

- [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025)
- [The Revolution of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2402.12451/) (2024)
- [OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding](https://mlsanity.com/paper/2406.19389/) (2024)
- [SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model](https://mlsanity.com/paper/2412.04292/) (2024)
- [PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model](https://mlsanity.com/paper/2403.14598/) (2024)
- [Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding](https://mlsanity.com/paper/2503.06287/) (2025)
- [SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation](https://mlsanity.com/paper/2409.10542/) (2024)
- [EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model](https://mlsanity.com/paper/2406.20076/) (2024)
- [Text4Seg: Reimagining Image Segmentation as Text Generation](https://mlsanity.com/paper/2410.09855/) (2024)
- [Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities](https://mlsanity.com/paper/2410.17385/) (2024)
- [LaSagnA: Language-based Segmentation Assistant for Complex Queries](https://mlsanity.com/paper/2404.08506/) (2024)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Bridging the Divide: Reconsidering Softmax and Linear Attention](https://mlsanity.com/paper/2412.06590/) (2024)
- [ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation](https://mlsanity.com/paper/2502.18364/) (2025)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [HyperSeg: Towards Universal Visual Segmentation with Large Language Model](https://mlsanity.com/paper/2411.17606/) (2024)
- [InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models](https://mlsanity.com/paper/2412.14006/) (2024)
- [SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories](https://mlsanity.com/paper/2503.08625/) (2025)
- [MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation](https://mlsanity.com/paper/2503.13881/) (2025)
- [UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface](https://mlsanity.com/paper/2503.01342/) (2025)
- [The Devil is in Temporal Token: High Quality Video Reasoning Segmentation](https://mlsanity.com/paper/2501.08549/) (2025)
- [Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators](https://mlsanity.com/paper/2408.05710/) (2024)
- [POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentation](https://mlsanity.com/paper/2504.00640/) (2025)
- [Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding](https://mlsanity.com/paper/2504.10465/) (2025)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model](https://mlsanity.com/paper/2503.13026/) (2025)
- [ViT3: Unlocking Test-Time Training in Vision](https://mlsanity.com/paper/2512.01643/) (2025)
- [SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes](https://mlsanity.com/paper/2506.01558/) (2025)
- [MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output](https://mlsanity.com/paper/2510.10011/) (2025)

## Abstract (arXiv)

Generalized Referring Expression Segmentation (GRES) extends the scope of classic RES to refer to multiple objects in one expression or identify the empty targets absent in the image. GRES poses challenges in modeling the complex spatial relationships of the instances in the image and identifying non-existing referents. Multimodal Large Language Models (MLLMs) have recently shown tremendous progress in these complicated vision-language tasks. Connecting Large Language Models (LLMs) and vision models, MLLMs are proficient in understanding contexts with visual inputs. Among them, LISA, as a representative, adopts a special [SEG] token to prompt a segmentation mask decoder, e.g., SAM, to enable MLLMs in the RES task. However, existing solutions to GRES remain unsatisfactory since current segmentation MLLMs cannot correctly handle the cases where users might reference multiple subjects in a singular prompt or provide descriptions incongruent with any image target. In this paper, we propose Generalized Segmentation Vision Assistant (GSVA) to address this gap. Specifically, GSVA reuses the [SEG] token to prompt the segmentation model towards supporting multiple mask references simultaneously and innovatively learns to generate a [REJ] token to reject the null targets explicitly. Experiments validate GSVA's efficacy in resolving the GRES issue, marking a notable enhancement and setting a new record on the GRES benchmark gRefCOCO dataset. GSVA also proves effective across various classic referring segmentation and comprehension tasks.