mlsanityData to 7 Oct 2026

Image Segmentation Using Text and Image Prompts

arXiv 2112.10003 ↗First posted 18 Dec 2021cs.CVcited by 509

Not on the map. Why?

Cited by 102 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Image segmentation is usually addressed by training a model for a fixed set of object classes. Incorporating additional classes or more complex queries later is expensive as it requires re-training the model on a dataset that encompasses these expressions. Here we propose a system that can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text or an image. This approach enables us to create a unified model (trained once) for three common segmentation tasks, which come with distinct challenges: referring expression segmentation, zero-shot segmentation and one-shot segmentation. We build upon the CLIP model as a backbone which we extend with a transformer-based decoder that enables dense prediction. After training on an extended version of the PhraseCut dataset, our system generates a binary segmentation map for an image based on a free-text prompt or on an additional image expressing the query. We analyze different variants of the latter image-based prompts in detail. This novel hybrid input allows for dynamic adaptation not only to the three segmentation tasks mentioned above, but to any binary segmentation task where a text or image query can be formulated. Finally, we find our system to adapt well to generalized queries involving affordances or properties. Code is available at https://eckerlab.org/code/clipseg.

Key facts. Cited by 509 papers in the map (data through 7 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 18 Dec 2021.

Builds on

42 of 59 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  11. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
30 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  3. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  4. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  5. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  6. 2016Modeling Context in Referring Expressionscited by 1,054
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  8. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  9. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  10. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2017Learning multiple visual domains with residual adapterscited by 553
  13. 2019PANet: Few-Shot Image Semantic Segmentation with Prototype Alignmentcited by 449
  14. 2017One-Shot Learning for Semantic Segmentationcited by 337
  15. 2020Prior Guided Feature Enrichment Network for Few-Shot Segmentationcited by 271
  16. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  17. 2016Segmentation from Natural Language Expressionscited by 257
  18. 2019Zero-Shot Semantic Segmentationcited by 245
  19. 2019CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learningcited by 217
  20. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  21. 2021Hypercorrelation Squeeze for Few-Shot Segmentationcited by 193
  22. 2020Prototype Mixture Models for Few-shot Semantic Segmentationcited by 190
  23. 2020Part-aware Prototype Network for Few-shot Semantic Segmentationcited by 162
  24. 2020Few-Shot Segmentation Without Meta-Learning: A Good Transductive Inference Is All You Need?cited by 135
  25. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  26. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  27. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  28. 2021Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentationcited by 53
  29. 2018One-Shot Instance Segmentationcited by 43
  30. 2018Few-Shot Segmentation Propagation with Guided Networkscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 124; 2024: 155; 2025: 123; 2026: 91.

509 papers.

Sort:

By area:

  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  4. 2023LERF: Language Embedded Radiance Fieldscited by 538
  5. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  6. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  7. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  8. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  9. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  10. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  11. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  12. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 233
38 more
  1. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  2. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  3. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  4. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  5. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  6. 2022Diffusion-based Image Translation using Disentangled Style and Content Representationcited by 148
  7. 2023LEDITS++: Limitless Image Editing using Text-to-Image Modelscited by 147
  8. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  9. 2023Diffuse, Attend, and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusioncited by 131
  10. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  11. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 115
  12. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  13. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  14. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  15. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  16. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 92
  17. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  18. 2024FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understandingcited by 86
  19. 2023CompoDiff: Versatile Composed Image Retrieval With Latent Diffusioncited by 84
  20. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  21. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  22. 2023Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignmentcited by 71
  23. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  24. 2023Inst-Inpaint: Instructing to Remove Objects with Diffusion Modelscited by 66
  25. 2023Zero-shot spatial layout conditioning for text-to-image diffusion modelscited by 64
  26. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  27. 2023SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Modelcited by 61
  28. 2023Segment Any Anomaly without Training via Hybrid Prompt Regularizationcited by 59
  29. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  30. 2024Gen3DSR: Generalizable 3D Scene Reconstruction via Divide and Conquer from a Single Viewcited by 59
  31. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  32. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  33. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoningcited by 52
  34. 2022X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusioncited by 49
  35. 2024Towards Understanding the Working Mechanism of Text-to-Image Diffusion Modelcited by 48
  36. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  37. 2024X-VILA: Cross-Modality Alignment for Large Language Modelcited by 46
  38. 2022Shape-Guided Diffusion with Inside-Outside Attentioncited by 45

50 most cited of 509 All 509 (JSON)