mlsanityData to 14 Oct 2026

Synthetic Captions for Open-Vocabulary Zero-Shot Segmentation

arXiv 2509.11840 ↗First posted 15 Sep 2025cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Generative vision-language models (VLMs) exhibit strong high-level image understanding but lack spatially dense alignment between vision and language modalities, as our findings indicate. Orthogonal to advancements in generative VLMs, another line of research has focused on representation learning for vision-language alignment, targeting zero-shot inference for dense tasks like segmentation. In this work, we bridge these two directions by densely aligning images with synthetic descriptions generated by VLMs. Synthetic captions are inexpensive, scalable, and easy to generate, making them an excellent source of high-level semantic understanding for dense alignment methods. Empirically, our approach outperforms prior work on standard zero-shot open-vocabulary segmentation benchmarks/datasets, while also being more data-efficient.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 48 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

31 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining30 Apr

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
18 more
  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  2. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  3. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  5. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  6. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  7. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  8. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  9. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  10. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  11. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  12. 2022Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learningcited by 115
  13. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  14. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  15. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  16. 2023CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Freecited by 39
  17. 2023Zero-guidance Segmentation Using Zero Segment Labelscited by 18
  18. 2024A Simple Framework for Open-Vocabulary Zero-Shot Segmentationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Unlocking Diffusion Hierarchies: Adaptive Timestep Selection for Zero-Shot Segmentationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Unifying Vision-Language Latents for Zero-label Image Caption Enhancement
  2. 2024Synth2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings
  3. 2024Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
  4. 2025Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model
  5. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries
  6. 2024S3: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Models
  7. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity
  8. 2024Grounding Descriptions in Images informs Zero-Shot Visual Recognition