# Probabilistic Prototype Calibration of Vision-Language Models for Generalized Few-shot Semantic Segmentation

arXiv: https://arxiv.org/abs/2506.22979 · HTML: https://mlsanity.com/paper/2506.22979/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 28 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020)
- [Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2108.04024/) (2021)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning](https://mlsanity.com/paper/1903.02351/) (2019)
- [Prototype Mixture Models for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2008.03898/) (2020)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [Learning What Not to Segment: A New Perspective on Few-Shot Segmentation](https://mlsanity.com/paper/2203.07615/) (2022)
- [Hierarchical Dense Correlation Distillation for Few-Shot Segmentation](https://mlsanity.com/paper/2303.14652/) (2023)
- [Generalized Few-shot Semantic Segmentation](https://mlsanity.com/paper/2010.05210/) (2020)
- [Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2108.06536/) (2021)
- [Bayesian Prompt Learning for Image-Language Model Generalization](https://mlsanity.com/paper/2210.02390/) (2022)
- [Dynamic Prototype Convolution Network for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2204.10638/) (2022)
- [SemiVL: Semi-Supervised Semantic Segmentation with Vision-Language Guidance](https://mlsanity.com/paper/2311.16241/) (2023)
- [Prototype-based Incremental Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2012.01415/) (2020)

## Built on by (most cited first, 1 of 1)

- [Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models](https://mlsanity.com/paper/2508.01225/) (2025)

## Abstract (arXiv)

Generalized Few-Shot Semantic Segmentation (GFSS) aims to extend a segmentation model to novel classes with only a few annotated examples while maintaining performance on base classes. Recently, pretrained vision-language models (VLMs) such as CLIP have been leveraged in GFSS to improve generalization on novel classes through multi-modal prototypes learning. However, existing prototype-based methods are inherently deterministic, limiting the adaptability of learned prototypes to diverse samples, particularly for novel classes with scarce annotations. To address this, we propose FewCLIP, a probabilistic prototype calibration framework over multi-modal prototypes from the pretrained CLIP, thus providing more adaptive prototype learning for GFSS. Specifically, FewCLIP first introduces a prototype calibration mechanism, which refines frozen textual prototypes with learnable visual calibration prototypes, leading to a more discriminative and adaptive representation. Furthermore, unlike deterministic prototype learning techniques, FewCLIP introduces distribution regularization over these calibration prototypes. This probabilistic formulation ensures structured and uncertainty-aware prototype learning, effectively mitigating overfitting to limited novel class data while enhancing generalization. Extensive experimental results on PASCAL-5i and COCO-20i datasets demonstrate that our proposed FewCLIP significantly outperforms state-of-the-art approaches across both GFSS and class-incremental setting. The code is available at https://github.com/jliu4ai/FewCLIP.