mlsanityData to 7 Oct 2026

GroupViT: Semantic Segmentation Emerges from Text Supervision

arXiv 2202.11094 ↗First posted 22 Feb 2022cs.CVcited by 529

Not on the map. Why?

Cited by 97 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision. We open-source our code at https://github.com/NVlabs/GroupViT .

Key facts. Cited by 529 papers in the map (data through 7 Oct 2026). Builds on 79 of its 103 references that are in the map. First posted on arXiv 22 Feb 2022.

Builds on

79 of 103 references.

References · 79

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2016Layer Normalizationcited by 5,579
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  12. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2016Pyramid Scene Parsing Networkcited by 2,472
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  12. 2021ViViT: A Video Vision Transformercited by 1,610
  13. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  14. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  15. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  16. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  17. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  18. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  19. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  20. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  21. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  22. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  23. 2021Video Swin Transformercited by 1,090
  24. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  25. 2021Multiscale Vision Transformerscited by 896
  26. 2020Object-Centric Learning with Slot Attentioncited by 865
  27. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  28. 2021Going deeper with Image Transformerscited by 759
  29. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  30. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  31. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  32. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  33. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  34. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  35. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  36. 2018Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentationcited by 321
  37. 2020Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentationcited by 299
  38. 2019Variational Adversarial Active Learningcited by 284

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 43; 2023: 156; 2024: 146; 2025: 106; 2026: 78.

529 papers.

Sort:

By area:

  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  4. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  5. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  6. 2023Segment Everything Everywhere All at Oncecited by 546
  7. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  8. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  9. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  10. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  11. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  12. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
38 more
  1. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  2. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  3. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 276
  4. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 219
  5. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  6. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  7. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  8. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  9. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  10. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 165
  11. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  12. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  13. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  14. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  15. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  16. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  17. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  18. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  19. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  20. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  21. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 115
  22. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  23. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  24. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  25. 2022LViT: Language meets Vision Transformer in Medical Image Segmentationcited by 110
  26. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  27. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  28. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 95
  29. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 95
  30. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  31. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  32. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 89
  33. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  34. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  35. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  36. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  37. 2023TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integrationcited by 76
  38. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76

50 most cited of 529 All 529 (JSON)