mlsanityData to 14 Oct 2026

Unified Open-World Segmentation with Multi-Modal Prompts

arXiv 2510.10524 ↗First posted 12 Oct 2025cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

In this work, we present COSINE, a unified open-world segmentation model that consolidates open-vocabulary segmentation and in-context segmentation with multi-modal prompts (e.g., text and image). COSINE exploits foundation models to extract representations for an input image and corresponding multi-modal prompts, and a SegDecoder to align these representations, model their interaction, and obtain masks specified by input prompts across different granularities. In this way, COSINE overcomes architectural discrepancies, divergent learning objectives, and distinct representation learning strategies of previous pipelines for open-vocabulary segmentation and in-context segmentation. Comprehensive experiments demonstrate that COSINE has significant performance improvements in both open-vocabulary and in-context segmentation tasks. Our exploratory analyses highlight that the synergistic collaboration between using visual and textual prompts leads to significantly improved generalization over single-modality approaches.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 65 of its 74 references that are in the map. First posted on arXiv 12 Oct 2025.

Builds on

65 of 74 references.

References · 65

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023Segment Anythingcited by 8,234
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2017Mask R-CNNcited by 7,277
  11. 2022Scalable Diffusion Models with Transformerscited by 6,054
  12. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
38 more
  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2022A ConvNet for the 2020scited by 3,579
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2016Pyramid Scene Parsing Networkcited by 2,479
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  9. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  10. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  13. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  14. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  15. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  16. 2019Panoptic Feature Pyramid Networkscited by 645
  17. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 596
  18. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  19. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  20. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  21. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  22. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 383
  23. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  24. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  25. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  26. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  27. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  28. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  29. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  30. 2023Personalize Segment Anything Model with One Shotcited by 250
  31. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  32. 2021Associating Objects with Transformers for Video Object Segmentationcited by 226
  33. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  34. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  35. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  36. 2023SegGPT: Segmenting Everything In Contextcited by 194
  37. 2021Hypercorrelation Squeeze for Few-Shot Segmentationcited by 193
  38. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 192

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?cited by 0
  2. 2026HERO: Hierarchical Embedding-Refinement for Open-Vocabulary Temporal Sentence Grounding in Videoscited by 0
  3. 2026UniSpector: Towards Universal Open-set Defect Recognition via Spectral-Contrastive Visual Promptingcited by 0
  4. 2026Robust Promptable Video Object Segmentationcited by 0
  5. 2026Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentationcited by 0
  6. 2026Training-Free Entity-Level Few-Shot Segmentation of Remote Sensing Images with Advection Refinementcited by 0
  7. 2026TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervisioncited by 0