mlsanityData to 14 Oct 2026

SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation

arXiv 2409.10542 ↗First posted 1 Sep 2024cs.AI cs.CL cs.CVcited by 69

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce SAM4MLLM, an innovative approach which integrates the Segment Anything Model (SAM) with Multi-Modal Large Language Models (MLLMs) for pixel-aware tasks. Our method enables MLLMs to learn pixel-level location information without requiring excessive modifications to the existing model architecture or adding specialized tokens. We introduce an inquiry-based approach that can effectively find prompt points for SAM to perform segmentation based on MLLM. It combines detailed visual information with the powerful expressive capabilities of large language models in a unified language-based manner without additional computational overhead in learning. Experimental results on pubic benchmarks demonstrate the effectiveness of our approach.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 66 references that are in the map. First posted on arXiv 1 Sep 2024.

Builds on

60 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023GSVA: Generalized Segmentation via Multimodal Large Language Models15 Dec
  2. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  3. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  5. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Segment Anythingcited by 8,234
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2017Mask R-CNNcited by 7,277
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023Mistral 7Bcited by 3,164
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  10. 2016Modeling Context in Referring Expressionscited by 1,063
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  13. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  14. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  15. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  16. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  17. 2023Segment Everything Everywhere All at Oncecited by 548
  18. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  19. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  20. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  21. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  22. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  23. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  24. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  25. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  26. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  27. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  28. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  29. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  30. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 232
  31. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 148
  32. 2023PACO: Parts and Attributes of Common Objectscited by 146
  33. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 139
  34. 2022Panoptic Scene Graph Generationcited by 114
  35. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  36. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  37. 2021PartImageNet: A Large, High-Quality Dataset of Partscited by 107
  38. 2020Linguistic Structure Guided Context Modeling for Referring Image Segmentationcited by 102

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 37; 2026: 30.

69 papers.

Sort:

By area:

  1. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
  2. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  3. 2025SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learningcited by 31
  4. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 30
  5. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  6. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  7. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  8. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  9. 2025Multimodal Referring Segmentation: A Surveycited by 22
  10. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 20
  11. 2025DeRIS: Decoupling Perception and Cognition for Enhanced Referring Image Segmentation through Loopback Synergycited by 13
  12. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
38 more
  1. 2025Reasoning Segmentation for Images and Videos: A Surveycited by 12
  2. 2025CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for Multi-Organ Segmentationcited by 12
  3. 2025EIAD: Explainable Industrial Anomaly Detection Via Multi-Modal Large Language Modelscited by 11
  4. 2026SAMTok: Representing Any Mask with Two Wordscited by 11
  5. 2025PropVG: End-to-End Proposal-Driven Visual Grounding with Multi-Granularity Discriminationcited by 10
  6. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 9
  7. 2025ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generationcited by 6
  8. 2025Refer to Any Segmentation Mask Group With Vision-Language Promptscited by 5
  9. 2025LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistancecited by 5
  10. 2025CoT Referring: Improving Referring Expression Tasks with Grounded Reasoningcited by 5
  11. 2025Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMscited by 5
  12. 2025ARGenSeg: Image Segmentation with Autoregressive Image Generation Modelcited by 5
  13. 2026Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Groundingcited by 5
  14. 2025LGD: Leveraging Generative Descriptions for Zero-Shot Referring Image Segmentationcited by 4
  15. 2025RESAnything: Attribute Prompting for Arbitrary Referring Segmentationcited by 4
  16. 2025Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentationcited by 4
  17. 2025Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Promptingcited by 4
  18. 2026GenMask: Adapting DiT for Segmentation via Direct Mask Generationcited by 3
  19. 2025HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentationcited by 2
  20. 2026Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrievalcited by 2
  21. 2026Enabling Training-Free Text-Based Remote Sensing Segmentationcited by 2
  22. 2026Rethinking MLLM Itself as a Segmenter with a Single Segmentation Tokencited by 2
  23. 2026SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Predictioncited by 2
  24. 2025SynRES: Towards Referring Expression Segmentation in the Wild via Synthetic Datacited by 1
  25. 2025RIS-LAD: A Benchmark and Model for Referring Low-Altitude Drone Image Segmentationcited by 1
  26. 2025PixelArena: A benchmark for Pixel-Precision Visual Intelligencecited by 1
  27. 2026From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMscited by 1
  28. 2026Tarot-SAM3: Training-free SAM3 for Any Referring Expression Segmentationcited by 1
  29. 2026Vision Harnessing Agent for Open Ad-hoc Segmentationcited by 1
  30. 2026FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentationcited by 1
  31. 2026Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPOcited by 1
  32. 2026Adapting Open-Weight MLLMs to Generate Point Prompts for Electron Microscopy Segmentationcited by 1
  33. 2024Phrase-Instance Alignment for Generalized Referring Segmentationcited by 0
  34. 2025R2SM: Referring and Reasoning for Selective Maskscited by 0
  35. 2025Decoupling the Image Perception and Multimodal Reasoning for Reasoning Segmentation with Digital Twin Representationscited by 0
  36. 2025Proxy-Embedding as an Adversarial Teacher: An Embedding-Guided Bidirectional Attack for Referring Expression Segmentation Modelscited by 0
  37. 2025MedPrompt: LLM-CNN Fusion with Weight Routing for Medical Image Segmentation and Classificationcited by 0
  38. 2025ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large Vision-and-Language Modelscited by 0

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder
  2. 2026ResAgent: Entropy-based Prior Point Discovery and Visual Reasoning for Referring Expression Segmentation
  3. 2025Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation
  4. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model
  5. 2024Empowering Segmentation Ability to Multi-modal Large Language Models
  6. 2025X-SAM: From Segment Anything to Any Segmentation
  7. 2026VespaSeg: A Resource-Aware Ground-then-Segment Pipeline for Referring Expression Segmentation ↗
  8. 2026An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation