mlsanityData to 14 Oct 2026

Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation

arXiv 2404.11998 ↗First posted 18 Apr 2024cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Referring image segmentation (RIS) aims to precisely segment referents in images through corresponding natural language expressions, yet relying on cost-intensive mask annotations. Weakly supervised RIS thus learns from image-text pairs to pixel-level semantics, which is challenging for segmenting fine-grained masks. A natural approach to enhancing segmentation precision is to empower weakly supervised RIS with the image segmentation foundation model SAM. Nevertheless, we observe that simply integrating SAM yields limited benefits and can even lead to performance regression due to the inevitable noise issues and challenges in excessive focus on object parts. In this paper, we present an innovative framework, Point PrompTing (PPT), incorporated with the proposed multi-source curriculum learning strategy to address these challenges. Specifically, the core of PPT is a point generator that not only harnesses CLIP's text-image alignment capability and SAM's powerful mask generation ability but also generates negative point prompts to address the noisy and excessive focus issues inherently and effectively. In addition, we introduce a curriculum learning strategy with object-centric images to help PPT gradually learn from simpler yet precise semantic alignment to more complex RIS. Experiments demonstrate that our PPT significantly and consistently outperforms prior weakly supervised techniques on mIoU by 11.34%, 14.14%, and 6.97% across RefCOCO, RefCOCO+, and G-Ref, respectively.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 55 of its 78 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

55 of 78 references.

References · 55

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2023Segment Anythingcited by 8,234
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  5. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  6. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  10. 2016Modeling Context in Referring Expressionscited by 1,063
  11. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  13. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  14. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  15. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 488
  16. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  17. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  18. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  19. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  20. 2016Segmentation from Natural Language Expressionscited by 257
  21. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  22. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  23. 2019Dice Loss for Data-imbalanced NLP Taskscited by 182
  24. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 150
  25. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  26. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 120
  27. 2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
  28. 2019Dynamic Graph Attention for Referring Expression Comprehensioncited by 111
  29. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  30. 2018Dynamic Multimodal Instance Segmentation guided by natural language queriescited by 83
  31. 2020Multi-Task Curriculum Framework for Open-Set Semi-Supervised Learningcited by 82
  32. 2023Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentationcited by 61
  33. 2020Graph-Structured Referring Expression Reasoning in The Wildcited by 58
  34. 2023Contrastive Grouping with Transformer for Referring Image Segmentationcited by 54
  35. 2023EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignmentcited by 44
  36. 2023SAMAug: Point Prompt Augmentation for Segment Anything Modelcited by 38
  37. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  38. 2023Referring Image Segmentation Using Text Supervisioncited by 35

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 15; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  2. 2025Multimodal Referring Segmentation: A Surveycited by 22
  3. 2025Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentationcited by 14
  4. 2025Rethinking Query-based Transformer for Continual Image Segmentationcited by 14
  5. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  6. 2024Boosting Weakly-Supervised Referring Image Segmentation via Progressive Comprehensioncited by 10
  7. 2025Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Contextcited by 10
  8. 2025WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentationcited by 9
  9. 2025Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EMcited by 8
  10. 2025Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formatscited by 6
  11. 2025LGD: Leveraging Generative Descriptions for Zero-Shot Referring Image Segmentationcited by 4
  12. 2025Sim-DETR: Unlock DETR for Temporal Sentence Groundingcited by 4
11 more
  1. 2025Closed-Loop Transfer for Weakly-supervised Affordance Groundingcited by 4
  2. 2025Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challengescited by 3
  3. 2025No More Sibling Rivalry: Debiasing Human-Object Interaction Detectioncited by 3
  4. 2025RefAM: Attention Magnets for Zero-Shot Referral Segmentationcited by 2
  5. 2026SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentationcited by 2
  6. 2025Augmenting Moment Retrieval: Zero-Dependency Two-Stage Learningcited by 1
  7. 2026Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentationcited by 1
  8. 2024The ART of Composition: Attention-Regularized Training for Compositional Visual Groundingcited by 0
  9. 2025Understanding What Is Not Said:Referring Remote Sensing Image Segmentation with Scarce Expressionscited by 0
  10. 2026Weakly-Supervised Referring Video Object Segmentation through Text Supervisioncited by 0
  11. 2026CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generationcited by 0