mlsanityData to 7 Oct 2026

EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model

arXiv 2406.20076 ↗First posted 28 Jun 2024cs.CVcited by 63

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Segment Anything Model (SAM) has attracted widespread attention for its superior interactive segmentation capabilities with visual prompts while lacking further exploration of text prompts. In this paper, we empirically investigate what text prompt encoders (e.g., CLIP or LLM) are good for adapting SAM for referring expression segmentation and introduce the Early Vision-language Fusion-based SAM (EVF-SAM). EVF-SAM is a simple yet effective referring segmentation method which exploits multimodal prompts (i.e., image and text) and comprises a pre-trained vision-language model to generate referring prompts and a SAM model for segmentation. Surprisingly, we observe that: (1) multimodal prompts and (2) vision-language models with early fusion (e.g., BEIT-3) are beneficial for prompting SAM for accurate referring segmentation. Our experiments show that the proposed EVF-SAM based on BEIT-3 can obtain state-of-the-art performance on RefCOCO/+/g for referring expression segmentation and demonstrate the superiority of prompting SAM with early vision-language fusion. In addition, the proposed EVF-SAM with 1.32B parameters achieves remarkably higher performance while reducing nearly 82% of parameters compared to previous SAM methods based on large multimodal models.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 67 of its 86 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

67 of 86 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Unveiling Encoder-Free Vision-Language Models17 Jun
  2. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model21 Mar
  3. 2023GSVA: Generalized Segmentation via Multimodal Large Language Models15 Dec
  4. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  5. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  7. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug

Other references · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
38 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  3. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  4. 2016Modeling Context in Referring Expressionscited by 1,054
  5. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  6. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  7. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  8. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  9. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  10. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  11. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  12. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  13. 2023Segment Anything in High Qualitycited by 373
  14. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  15. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  16. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 345
  17. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  18. 2023Fast Segment Anythingcited by 286
  19. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  20. 2016Segmentation from Natural Language Expressionscited by 257
  21. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  22. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  23. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  24. 2023Inpaint Anything: Segment Anything Meets Image Inpaintingcited by 233
  25. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 230
  26. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  27. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  28. 2019FSS-1000: A 1000-Class Dataset for Few-Shot Segmentationcited by 160
  29. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  30. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  31. 2023PACO: Parts and Attributes of Common Objectscited by 146
  32. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  33. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  34. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  35. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  36. 2021PartImageNet: A Large, High-Quality Dataset of Partscited by 107
  37. 2024VRP-SAM: SAM with Visual Reference Promptcited by 75
  38. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 38; 2026: 17.

63 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  2. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
  3. 2024SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentationcited by 63
  4. 2024SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentationcited by 43
  5. 2024Segment Anything in Medical Images and Videos: Benchmark and Deploymentcited by 42
  6. 2025SegEarth-R1: Geospatial Pixel Reasoning via Large Language Modelcited by 40
  7. 2025Multimodal Referring Segmentation: A Surveycited by 22
  8. 2024Efficient Track Anythingcited by 20
  9. 2024Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentationcited by 18
  10. 2025LENS: Learning to Segment Anything with Unified Reinforced Reasoningcited by 18
  11. 2025RSRefSeg 2: Decoupling Referring Remote Sensing Image Segmentation with Foundation Modelscited by 17
  12. 2025SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenescited by 16
38 more
  1. 2025OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Promptscited by 14
  2. 2025SAM2 for Image and Video Segmentation: A Comprehensive Surveycited by 13
  3. 2025GroundingSuite: Measuring Complex Multi-Granular Pixel Groundingcited by 10
  4. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
  5. 2025VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learningcited by 8
  6. 2025MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentationcited by 7
  7. 2025AffordanceSAM: Segment Anything Once More in Affordance Groundingcited by 6
  8. 2025Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Optical Remote Sensingcited by 6
  9. 2026RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulationcited by 6
  10. 2026InstructSAM: Segment Any Instance with Any Instructionscited by 6
  11. 2025Zoomer: Adaptive Image Focus Optimization for Black-box MLLMcited by 5
  12. 2025RSAgent: Learning to Reason and Act for Text-Guided Segmentation via Multi-Turn Tool Invocationscited by 5
  13. 2025RS2-SAM2: Customized SAM2 for Referring Remote Sensing Image Segmentationcited by 4
  14. 2025RESAnything: Attribute Prompting for Arbitrary Referring Segmentationcited by 4
  15. 2025Lego-Edit: A General Image Editing Framework with Model-Level Bricks and MLLM Buildercited by 4
  16. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4
  17. 2025OmniGround: A Comprehensive Spatio-Temporal Grounding Benchmark for Real-World Complex Scenarioscited by 4
  18. 2025Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Promptingcited by 4
  19. 2025Find First, Track Next: Decoupling Identification and Propagation in Referring Video Object Segmentationcited by 3
  20. 2025SparseGS-W: Sparse-View 3D Gaussian Splatting in the Wild with Generative Priorscited by 3
  21. 2025RAU: Reference-based Anatomical Understanding with Vision Language Modelscited by 3
  22. 2026MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentationcited by 3
  23. 2025Beyond Blanket Masking: Examining Granularity for Privacy Protection in Images Captured by Blind and Low Vision Userscited by 2
  24. 2026Conversational Image Segmentation: Grounding Abstract Concepts with Scalable Supervisioncited by 2
  25. 2026SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentationcited by 2
  26. 2026SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Predictioncited by 2
  27. 2025Audio Visual Segmentation Through Text Embeddingscited by 1
  28. 2025ACCORD: Alleviating Concept Coupling through Dependence Regularization for Text-to-Image Diffusion Personalizationcited by 1
  29. 2025BiPrompt-SAM: Enhancing Image Segmentation via Explicit Selection between Point and Text Promptscited by 1
  30. 2025SynRES: Towards Referring Expression Segmentation in the Wild via Synthetic Datacited by 1
  31. 2025SAM-PTx: Text-Guided Fine-Tuning of SAM with Parameter-Efficient, Parallel-Text Adapterscited by 1
  32. 2025ReLAM: Learning Anticipation Model for Rewarding Visual Robotic Manipulationcited by 1
  33. 2025LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentationcited by 1
  34. 2025RobotSeg: A Model and Dataset for Segmenting Robots in Image and Videocited by 1
  35. 2026Vision Harnessing Agent for Open Ad-hoc Segmentationcited by 1
  36. 2026Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPOcited by 1
  37. 2026CROSS: Cascaded Distillation and Dual-Constraint Grounding for Remote Sensing Referring Segmentationcited by 1
  38. 2024Domain-invariant Representation Learning via Segment Anything Model for Blood Cell Classificationcited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation
  2. 2025Vision and Language Reference Prompt into SAM for Few-shot Segmentation
  3. 2026CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation
  4. 2025Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation
  5. 2025X-SAM: From Segment Anything to Any Segmentation
  6. 2025Refer to Any Segmentation Mask Group With Vision-Language Prompts
  7. 2025Deformable Attentive Visual Enhancement for Referring Segmentation Using Vision-Language Model
  8. 2026GenSeg-R1: RL-Driven Vision-Language Grounding for Fine-Grained Referring Segmentation