mlsanityData to 7 Oct 2026

Weakly-supervised segmentation of referring expressions

arXiv 2205.04725 ↗First posted 10 May 2022cs.CV cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Visual grounding localizes regions (boxes or segments) in the image corresponding to given referring expressions. In this work we address image segmentation from referring expressions, a problem that has so far only been addressed in a fully-supervised setting. A fully-supervised setup, however, requires pixel-wise supervision and is hard to scale given the expense of manual annotation. We therefore introduce a new task of weakly-supervised image segmentation from referring expressions and propose Text grounded semantic SEGgmentation (TSEG) that learns segmentation masks directly from image-level referring expressions without pixel-level annotations. Our transformer-based method computes patch-text similarities and guides the classification objective during training with a new multi-label patch assignment mechanism. The resulting visual grounding model segments image regions corresponding to given natural language expressions. Our approach TSEG demonstrates promising results for weakly-supervised referring expression segmentation on the challenging PhraseCut and RefCOCO datasets. TSEG also shows competitive performance when evaluated in a zero-shot setting for semantic segmentation on Pascal VOC.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 55 of its 69 references that are in the map. First posted on arXiv 10 May 2022.

Builds on

55 of 69 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2017Mask R-CNNcited by 7,259
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
38 more
  1. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  5. 2021ViViT: A Video Vision Transformercited by 1,610
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  8. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  9. 2016Modeling Context in Referring Expressionscited by 1,054
  10. 2016Deep Networks with Stochastic Depthcited by 973
  11. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  13. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  14. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  15. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  16. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  17. 2018Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentationcited by 321
  18. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  19. 2016Segmentation from Natural Language Expressionscited by 257
  20. 2019Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relationscited by 245
  21. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  22. 2017Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approachcited by 201
  23. 2016Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentationcited by 198
  24. 2015Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentationcited by 189
  25. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  26. 2018Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi- Supervised Semantic Segmentationcited by 145
  27. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
  28. 2021Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentationcited by 127
  29. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  30. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  31. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  32. 2014Salient Object Detection: A Discriminative Regional Feature Integration Approachcited by 89
  33. 2019Joint Learning of Saliency Detection and Weakly Supervised Semantic Segmentationcited by 57
  34. 2021Discriminative Region Suppression for Weakly-Supervised Semantic Segmentationcited by 50
  35. 2014Weakly Supervised Action Labeling in Videos Under Ordering Constraintscited by 48
  36. 2017Weakly-supervised Visual Grounding of Phrases with Linguistic Structurescited by 40
  37. 2016ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localizationcited by 39
  38. 2019Adaptive Reconstruction Network for Weakly Supervised Referring Expression Groundingcited by 37

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 4; 2025: 6; 2026: 1.

21 papers.

Sort:

By area:

  1. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  2. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  3. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  4. 2023Ref-Diff: Zero-shot Referring Image Segmentation with Generative Modelscited by 33
  5. 2025IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasiscited by 26
  6. 2023Shatter and Gather: Learning Referring Image Segmentation with Text Supervisioncited by 25
  7. 2024Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentationcited by 23
  8. 2025Multimodal Referring Segmentation: A Surveycited by 22
  9. 2023Text Augmented Spatial-aware Zero-shot Referring Image Segmentationcited by 21
  10. 2023Extending CLIP's Image-Text Alignment to Referring Image Segmentationcited by 19
  11. 2023MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentationcited by 18
  12. 2025Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentationcited by 14
9 more
  1. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  2. 2022Training Vision-Language Transformers from Captionscited by 11
  3. 2024Toward Interactive Regional Understanding in Vision-Large Language Modelscited by 9
  4. 2025WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentationcited by 9
  5. 2025Direct Visual Grounding by Directing Attention of Visual Tokenscited by 6
  6. 2026MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Datacited by 5
  7. 2023Segment, Select, Correct: A Framework for Weakly-Supervised Referring Segmentationcited by 3
  8. 2023Towards Omni-supervised Referring Expression Segmentationcited by 1
  9. 2025AlignCAT: Visual-Linguistic Alignment of Category and Attribute for Weakly Supervised Visual Groundingcited by 1