mlsanityData to 14 Oct 2026

Scaling Open-Vocabulary Image Segmentation with Image-Level Labels

arXiv 2112.12143 ↗First posted 22 Dec 2021cs.CVcited by 473

Not on the map. Why?

Cited by 83 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We design an open-vocabulary image segmentation model to organize an image into meaningful regions indicated by arbitrary texts. Recent works (CLIP and ALIGN), despite attaining impressive open-vocabulary classification accuracy with image-level caption labels, are unable to segment visual concepts with pixels. We argue that these models miss an important step of visual grouping, which organizes pixels into groups before learning visual-semantic alignments. We propose OpenSeg to address the above issue while still making use of scalable image-level supervision of captions. First, it learns to propose segmentation masks for possible organizations. Then it learns visual-semantic alignments by aligning each word in a caption to one or a few predicted masks. We find the mask representations are the key to support learning image segmentation from captions, making it possible to scale up the dataset and vocabulary sizes. OpenSeg significantly outperforms the recent open-vocabulary method of LSeg by +19.9 mIoU on PASCAL dataset, thanks to its scalability.

Key facts. Cited by 473 papers in the map (data through 14 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 22 Dec 2021.

Builds on

42 of 63 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020End-to-End Object Detection with Transformerscited by 7,414
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  8. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  12. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
30 more
  1. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  2. 2016Modeling Context in Referring Expressionscited by 1,063
  3. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  5. 2018Panoptic Segmentationcited by 802
  6. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  7. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 596
  8. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  9. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  10. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  11. 2020Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentationcited by 300
  12. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  13. 2016Segmentation from Natural Language Expressionscited by 257
  14. 2019Zero-Shot Semantic Segmentationcited by 245
  15. 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
  16. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  17. 2019Connecting Vision and Language with Localized Narrativescited by 226
  18. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  19. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  20. 2014From Captions to Visual Concepts and Backcited by 109
  21. 2020Contrastive Learning for Weakly Supervised Phrase Groundingcited by 102
  22. 2021Multi-Task Self-Training for Learning General Representationscited by 85
  23. 2021Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentationcited by 77
  24. 2017Open Vocabulary Scene Parsingcited by 76
  25. 2021Open-World Entity Segmentationcited by 74
  26. 2021Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentationcited by 53
  27. 2021Pseudo-mask Matters in Weakly-supervised Semantic Segmentationcited by 48
  28. 2021Puzzle-CAM: Improved localization via matching partial and full featurescited by 37
  29. 2016Convolutional Oriented Boundariescited by 25
  30. 2021Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examplescited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 120; 2024: 140; 2025: 115; 2026: 90.

473 papers.

Sort:

By area:

  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  3. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  4. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  5. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  6. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  7. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 219
  8. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  9. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
  10. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  11. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  12. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
38 more
  1. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  2. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  3. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
  4. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  5. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  6. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  7. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  8. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 97
  9. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 96
  10. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  11. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 95
  12. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 94
  13. 2024T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergycited by 94
  14. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93
  15. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  16. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  17. 2023Grounding Everything: Emerging Localization Properties in Vision-Language Transformerscited by 91
  18. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  19. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  20. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 82
  21. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  22. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 80
  23. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  24. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 77
  25. 2023Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmentercited by 76
  26. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 75
  27. 2025Detect Anything via Next Point Predictioncited by 73
  28. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  29. 2023Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignmentcited by 71
  30. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 71
  31. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  32. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  33. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 69
  34. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66
  35. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  36. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 64
  37. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  38. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62

50 most cited of 473 All 473 (JSON)