# A Good Foundation is Worth Many Labels: Label-Efficient Panoptic Segmentation

arXiv: https://arxiv.org/abs/2405.19035 · HTML: https://mlsanity.com/paper/2405.19035/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 37 references that are in the map.
- First posted on arXiv 29 May 2024.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels](https://mlsanity.com/paper/2203.03884/) (2022)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)
- [Cut and Learn for Unsupervised Object Detection and Instance Segmentation](https://mlsanity.com/paper/2301.11320/) (2023)
- [ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2106.05095/) (2021)
- [PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering](https://mlsanity.com/paper/2103.17070/) (2021)
- [CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2212.09506/) (2022)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes](https://mlsanity.com/paper/1611.08323/) (2016)
- [Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation](https://mlsanity.com/paper/2012.10782/) (2020)
- [Discovering Object Masks with Transformers for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2206.06363/) (2022)
- [A Survey on Label-efficient Deep Image Segmentation: Bridging the Gap between Weak Supervision and Dense Prediction](https://mlsanity.com/paper/2207.01223/) (2022)
- [Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision](https://mlsanity.com/paper/2108.07682/) (2021)
- [PhenoBench -- A Large Dataset and Benchmarks for Semantic Image Interpretation in the Agricultural Domain](https://mlsanity.com/paper/2306.04557/) (2023)
- [Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transport](https://mlsanity.com/paper/2308.01779/) (2023)
- [Perceiving the Invisible: Proposal-Free Amodal Panoptic Segmentation](https://mlsanity.com/paper/2205.14637/) (2022)

## Built on by (most cited first, 7 of 7)

- [Image Segmentation in Foundation Model Era: A Survey](https://mlsanity.com/paper/2408.12957/) (2024)
- [Bridging Perspectives: Foundation Model Guided BEV Maps for 3D Object Detection and Tracking](https://mlsanity.com/paper/2510.10287/) (2025)
- [Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2509.20107/) (2025)
- [Label-Efficient LiDAR Panoptic Segmentation](https://mlsanity.com/paper/2503.02372/) (2025)
- [Hyp2Former: Hierarchy-Aware Hyperbolic Embeddings for Open-Set Panoptic Segmentation](https://mlsanity.com/paper/2605.02580/) (2026)
- [DVPSFormer: Efficient Online Depth-aware Video Panoptic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2607.26165/) (2026)
- [Leveraging Vision-Based Point Cloud Map Priors for Camera-Based 3D Object Detection and Online Vectorized HD Mapping](https://mlsanity.com/paper/2609.26325/) (2026)

## Abstract (arXiv)

A key challenge for the widespread application of learning-based models for robotic perception is to significantly reduce the required amount of annotated training data while achieving accurate predictions. This is essential not only to decrease operating costs but also to speed up deployment time. In this work, we address this challenge for PAnoptic SegmenTation with fEw Labels (PASTEL) by exploiting the groundwork paved by visual foundation models. We leverage descriptive image features from such a model to train two lightweight network heads for semantic segmentation and object boundary detection, using very few annotated training samples. We then merge their predictions via a novel fusion module that yields panoptic maps based on normalized cut. To further enhance the performance, we utilize self-training on unlabeled images selected by a feature-driven similarity scheme. We underline the relevance of our approach by employing PASTEL to important robot perception use cases from autonomous driving and agricultural robotics. In extensive experiments, we demonstrate that PASTEL significantly outperforms previous methods for label-efficient segmentation even when using fewer annotations. The code of our work is publicly available at http://pastel.cs.uni-freiburg.de.