# ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistency

arXiv: https://arxiv.org/abs/2302.10307 · HTML: https://mlsanity.com/paper/2302.10307/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 31 Jan 2023.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training](https://mlsanity.com/paper/2005.00200/) (2020)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2208.12262/) (2022)
- [Beyond Fixation: Dynamic Window Visual Transformer](https://mlsanity.com/paper/2203.12856/) (2022)
- [Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples](https://mlsanity.com/paper/2112.03185/) (2021)
- [Distilled Dual-Encoder Model for Vision-Language Understanding](https://mlsanity.com/paper/2112.08723/) (2021)

## Built on by (most cited first, 30 of 51)

- [ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference](https://mlsanity.com/paper/2407.12442/) (2024)
- [SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding](https://mlsanity.com/paper/2310.15308/) (2023)
- [ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2408.04883/) (2024)
- [CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation](https://mlsanity.com/paper/2312.12359/) (2023)
- [DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment](https://mlsanity.com/paper/2412.16334/) (2024)
- [Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter](https://mlsanity.com/paper/2309.02773/) (2023)
- [CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor](https://mlsanity.com/paper/2312.07661/) (2023)
- [Diffusion Models for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2306.09316/) (2023)
- [Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation](https://mlsanity.com/paper/2404.06542/) (2024)
- [Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2411.19331/) (2024)
- [CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free](https://mlsanity.com/paper/2309.14289/) (2023)
- [In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2408.04961/) (2024)
- [Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentation](https://mlsanity.com/paper/2411.15869/) (2024)
- [Adaptive Bidirectional Displacement for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2405.00378/) (2024)
- [ResCLIP: Residual Attention for Training-free Dense Vision-language Inference](https://mlsanity.com/paper/2411.15851/) (2024)
- [OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusion](https://mlsanity.com/paper/2407.07844/) (2024)
- [Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2310.19001/) (2023)
- [DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut](https://mlsanity.com/paper/2406.02842/) (2024)
- [What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2306.15521/) (2023)
- [Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation](https://mlsanity.com/paper/2309.13505/) (2023)
- [Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation](https://mlsanity.com/paper/2404.04231/) (2024)
- [Improved baselines for vision-language pre-training](https://mlsanity.com/paper/2305.08675/) (2023)
- [Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation](https://mlsanity.com/paper/2408.15205/) (2024)
- [MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation](https://mlsanity.com/paper/2308.04829/) (2023)
- [CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2411.13836/) (2024)
- [Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models](https://mlsanity.com/paper/2311.17095/) (2023)
- [Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2404.00262/) (2024)
- [FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models](https://mlsanity.com/paper/2403.20105/) (2024)
- [iSeg: An Iterative Refinement-based Framework for Training-free Segmentation](https://mlsanity.com/paper/2409.03209/) (2024)
- [Training-Free Class Purification for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2508.00557/) (2025)

## Abstract (arXiv)

Recently, great success has been made in learning visual representations from text supervision, facilitating the emergence of text-supervised semantic segmentation. However, existing works focus on pixel grouping and cross-modal semantic alignment, while ignoring the correspondence among multiple augmented views of the same image. To overcome such limitation, we propose multi-View Consistent learning (ViewCo) for text-supervised semantic segmentation. Specifically, we first propose text-to-views consistency modeling to learn correspondence for multiple views of the same input image. Additionally, we propose cross-view segmentation consistency modeling to address the ambiguity issue of text supervision by contrasting the segment features of Siamese visual encoders. The text-to-views consistency benefits the dense assignment of the visual features by encouraging different crops to align with the same text, while the cross-view segmentation consistency modeling provides additional self-supervision, overcoming the limitation of ambiguous text supervision for segmentation masks. Trained with large-scale image-text data, our model can directly segment objects of arbitrary categories in a zero-shot manner. Extensive experiments show that ViewCo outperforms state-of-the-art methods on average by up to 2.9%, 1.6%, and 2.4% mIoU on PASCAL VOC2012, PASCAL Context, and COCO, respectively.