# ConformalSAM: Unlocking the Potential of Foundational Segmentation Models in Semi-Supervised Semantic Segmentation with Conformal Prediction

arXiv: https://arxiv.org/abs/2507.15803 · HTML: https://mlsanity.com/paper/2507.15803/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 21 Jul 2025.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision](https://mlsanity.com/paper/2106.01226/) (2021)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [Uncertainty Sets for Image Classifiers using Conformal Prediction](https://mlsanity.com/paper/2009.14193/) (2020)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](https://mlsanity.com/paper/2203.03884/) (2022)
- [RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model](https://mlsanity.com/paper/2306.16269/) (2023)
- [PseudoSeg: Designing Pseudo Labels for Semantic Segmentation](https://mlsanity.com/paper/2010.09713/) (2020)
- [Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2208.09910/) (2022)
- [Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning](https://mlsanity.com/paper/2110.05474/) (2021)
- [Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping](https://mlsanity.com/paper/2305.11003/) (2023)
- [Augmentation Matters: A Simple-yet-Effective Approach to Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2212.04976/) (2022)
- [Semi-supervised Semantic Segmentation with Prototype-based Consistency Regularization](https://mlsanity.com/paper/2210.04388/) (2022)
- [CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2306.04300/) (2023)
- [Conflict-Based Cross-View Consistency for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2303.01276/) (2023)
- [AllSpark: Reborn Labeled Features from Unlabeled in Transformer for Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2403.01818/) (2024)
- [Semi-Supervised Semantic Segmentation via Gentle Teaching Assistant](https://mlsanity.com/paper/2301.07340/) (2023)
- [Logic-induced Diagnostic Reasoning for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2308.12595/) (2023)
- [Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration](https://mlsanity.com/paper/2406.06818/) (2024)
- [Towards the Uncharted: Density-Descending Feature Perturbation for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2403.06462/) (2024)
- [Cross Prompting Consistency with Segment Anything Model for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2407.05416/) (2024)
- [Kandinsky Conformal Prediction: Efficient Calibration of Image Segmentation Algorithms](https://mlsanity.com/paper/2311.11837/) (2023)
- [The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks](https://mlsanity.com/paper/2405.08886/) (2024)
- [Direct Prediction Set Minimization via Bilevel Conformal Classifier Training](https://mlsanity.com/paper/2506.06599/) (2025)

## Built on by (most cited first, 4 of 4)

- [Provably Minimum-Length Conformal Prediction Sets for Ordinal Classification](https://mlsanity.com/paper/2511.16845/) (2025)
- [Wat3R: Underwater 3D Geometry Learning without Annotations](https://mlsanity.com/paper/2607.08772/) (2026)
- [Anatomically-aware conformal prediction for medical image segmentation with random walks](https://mlsanity.com/paper/2601.18997/) (2026)
- [Bridging the Gap between Labeled and Unlabeled Data via Unified Flow with Feature Memory Bank](https://mlsanity.com/paper/2608.16681/) (2026)

## Abstract (arXiv)

Pixel-level vision tasks, such as semantic segmentation, require extensive and high-quality annotated data, which is costly to obtain. Semi-supervised semantic segmentation (SSSS) has emerged as a solution to alleviate the labeling burden by leveraging both labeled and unlabeled data through self-training techniques. Meanwhile, the advent of foundational segmentation models pre-trained on massive data, has shown the potential to generalize across domains effectively. This work explores whether a foundational segmentation model can address label scarcity in the pixel-level vision task as an annotator for unlabeled images. Specifically, we investigate the efficacy of using SEEM, a Segment Anything Model (SAM) variant fine-tuned for textual input, to generate predictive masks for unlabeled data. To address the shortcomings of using SEEM-generated masks as supervision, we propose ConformalSAM, a novel SSSS framework which first calibrates the foundation model using the target domain's labeled data and then filters out unreliable pixel labels of unlabeled data so that only high-confidence labels are used as supervision. By leveraging conformal prediction (CP) to adapt foundation models to target data through uncertainty calibration, ConformalSAM exploits the strong capability of the foundational segmentation model reliably which benefits the early-stage learning, while a subsequent self-reliance training strategy mitigates overfitting to SEEM-generated masks in the later training stage. Our experiment demonstrates that, on three standard benchmarks of SSSS, ConformalSAM achieves superior performance compared to recent SSSS methods and helps boost the performance of those methods as a plug-in.