mlsanityData to 14 Oct 2026

CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation

arXiv 2303.11797 ↗First posted 21 Mar 2023cs.CVcited by 219

Not on the map. Why?

Cited by 40 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Open-vocabulary semantic segmentation presents the challenge of labeling each pixel within an image based on a wide range of text descriptions. In this work, we introduce a novel cost-based approach to adapt vision-language foundation models, notably CLIP, for the intricate task of semantic segmentation. Through aggregating the cosine similarity score, i.e., the cost volume between image and text embeddings, our method potently adapts CLIP for segmenting seen and unseen classes by fine-tuning its encoders, addressing the challenges faced by existing methods in handling unseen classes. Building upon this, we explore methods to effectively aggregate the cost volume considering its multi-modal nature of being established between image and text embeddings. Furthermore, we examine various methods for efficiently fine-tuning CLIP.

Key facts. Cited by 219 papers in the map (data through 14 Oct 2026). Builds on 82 of its 115 references that are in the map. First posted on arXiv 21 Mar 2023.

Builds on

82 of 115 references.

References · 82

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2017Mask R-CNNcited by 7,277
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2016Layer Normalizationcited by 5,669
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
38 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2018Group Normalizationcited by 1,524
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  12. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  13. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
  14. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  15. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  16. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  17. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  18. 2021Scaling Vision Transformerscited by 999
  19. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  20. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  21. 2018Panoptic Segmentationcited by 802
  22. 2021Robust fine-tuning of zero-shot modelscited by 797
  23. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  24. 2022Language-driven Semantic Segmentationcited by 717
  25. 2021Extract Free Dense Labels from CLIPcited by 583
  26. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
  27. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  28. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  29. 2018Pyramid Stereo Matching Networkcited by 489
  30. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  31. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  32. 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 360
  33. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  34. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  35. 2022FlowFormer: A Transformer Architecture for Optical Flowcited by 275
  36. 2019Zero-Shot Semantic Segmentationcited by 245
  37. 2019Group-wise Correlation Stereo Networkcited by 235
  38. 2019Connecting Vision and Language with Localized Narrativescited by 226

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 51; 2025: 73; 2026: 76.

219 papers.

Sort:

By area:

  1. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  2. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  4. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 118
  5. 2023SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentationcited by 97
  6. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  7. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  8. 2024Explore the Potential of CLIP for Training-Free Open Vocabulary Semantic Segmentationcited by 61
  9. 2023One-Shot Open Affordance Learning with Foundation Modelscited by 45
  10. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 44
  11. 2023SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidancecited by 41
  12. 2024Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentationcited by 39
38 more
  1. 2024In Defense of Lazy Visual Grounding for Open-Vocabulary Semantic Segmentationcited by 38
  2. 2023Unifying Correspondence, Pose and NeRF for Pose-Free Novel View Synthesis from Stereo Pairscited by 37
  3. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  4. 2023Collaborating Foundation Models for Domain Generalized Semantic Segmentationcited by 34
  5. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 34
  6. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  7. 2025DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perceptioncited by 33
  8. 2023OV-PARTS: Towards Open-Vocabulary Part Segmentationcited by 31
  9. 2024Search3D: Hierarchical Open-Vocabulary 3D Segmentationcited by 31
  10. 2025Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensingcited by 31
  11. 2024Cross-View Completion Models are Zero-shot Correspondence Estimatorscited by 30
  12. 2023AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregationcited by 29
  13. 2023Open-vocabulary object 6D pose estimationcited by 29
  14. 2025MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentationcited by 28
  15. 2024SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Imagescited by 25
  16. 2023What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentationcited by 24
  17. 2024Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentationcited by 24
  18. 2025Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformerscited by 24
  19. 2025Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentationcited by 22
  20. 2025Emergent Outlier View Rejection in Visual Geometry Grounded Transformerscited by 22
  21. 2024Textual Query-Driven Mask Transformer for Domain Generalized Segmentationcited by 21
  22. 2025DynamicEarth: How Far are We from Open-Vocabulary Change Detection?cited by 21
  23. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  24. 2025C3G: Learning Compact 3D Representations with 2K Gaussianscited by 21
  25. 2024Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentationcited by 20
  26. 2023FoodSAM: Any Food Segmentationcited by 19
  27. 2024Towards Open-Vocabulary Semantic Segmentation Without Semantic Labelscited by 19
  28. 2024Towards Open-Vocabulary Remote Sensing Image Semantic Segmentationcited by 19
  29. 2026Multiplayer Interactive World Models with Representation Autoencoderscited by 19
  30. 2024Open Panoramic Segmentationcited by 17
  31. 2024CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentationcited by 17
  32. 2024Effective SAM Combination for Open-Vocabulary Semantic Segmentationcited by 17
  33. 2024Language Driven Occupancy Predictioncited by 17
  34. 2024LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understandingcited by 17
  35. 2025SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Imagescited by 16
  36. 2025MARIS: Marine Open-Vocabulary Instance Segmentation with Geometric Enhancement and Semantic Alignmentcited by 15
  37. 2023Open-Vocabulary Camouflaged Object Segmentationcited by 14
  38. 2024UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understandingcited by 13

50 most cited of 219 All 219 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Global Knowledge Calibration for Fast Open-Vocabulary Segmentation
  2. 2025FA-Seg: A Fast and Accurate Diffusion-Based Method for Open-Vocabulary Segmentation
  3. 2023Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
  4. 2025From Open-Vocabulary to Vocabulary-Free Semantic Segmentation
  5. 2024Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation
  6. 2023CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation
  7. 2026HyperCLIP++: Fine-tuning CLIP forOpen-vocabulary Semantic Segmentation in Hyperbolic Space
  8. 2024Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation