mlsanityData to 7 Oct 2026

Towards Generalizable Referring Image Segmentation via Target Prompt and Visual Coherence

arXiv 2312.00452 ↗First posted 1 Dec 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Referring image segmentation (RIS) aims to segment objects in an image conditioning on free-from text descriptions. Despite the overwhelming progress, it still remains challenging for current approaches to perform well on cases with various text expressions or with unseen visual entities, limiting its further application. In this paper, we present a novel RIS approach, which substantially improves the generalization ability by addressing the two dilemmas mentioned above. Specially, to deal with unconstrained texts, we propose to boost a given expression with an explicit and crucial prompt, which complements the expression in a unified context, facilitating target capturing in the presence of linguistic style changes. Furthermore, we introduce a multi-modal fusion aggregation module with visual guidance from a powerful pretrained model to leverage spatial relations and pixel coherences to handle the incomplete target masks and false positive irregular clumps which often appear on unseen visual entities. Extensive experiments are conducted in the zero-shot cross-dataset settings and the proposed approach achieves consistent gains compared to the state-of-the-art, e.g., 4.15%, 5.45%, and 4.64% mIoU increase on RefCOCO, RefCOCO+ and ReferIt respectively, demonstrating its effectiveness. Additionally, the results on GraspNet-RIS show that our approach also generalizes well to new scenarios with large domain shifts.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 39 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

34 of 39 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2018Group Normalizationcited by 1,514
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  11. 2016Modeling Context in Referring Expressionscited by 1,054
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
22 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  3. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  4. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  5. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  6. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  7. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  8. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  9. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  10. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  11. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  12. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  13. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  14. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  15. 2023Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentationcited by 61
  16. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
  17. 2023Contrastive Grouping with Transformer for Referring Image Segmentationcited by 54
  18. 2017Language-Based Image Editing with Recurrent Attentive Modelscited by 53
  19. 2022GraftNet: Towards Domain Generalized Stereo Matching with a Broad-Spectrum and Task-Oriented Featurecited by 43
  20. 2022CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentationcited by 28
  21. 2022Position-Aware Contrastive Alignment for Referring Image Segmentationcited by 9
  22. 2022Embodied Referring Expression for Manipulation Question Answering in Interactive Environmentcited by 7

Built on by

Not cited yet.