# Self-Supervised Vision Transformers Are Efficient Segmentation Learners for Imperfect Labels

arXiv: https://arxiv.org/abs/2401.12535 · HTML: https://mlsanity.com/paper/2401.12535/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 17 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Builds on (most cited first, 14 of 14)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://mlsanity.com/paper/1412.7062/) (2014)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [Railroad is not a Train: Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2105.08965/) (2021)
- [Adaptive Early-Learning Correction for Segmentation from Noisy Annotations](https://mlsanity.com/paper/2110.03740/) (2021)
- [Tree Energy Loss: Towards Sparsely Annotated Semantic Segmentation](https://mlsanity.com/paper/2203.10739/) (2022)
- [Scribble-Supervised Semantic Segmentation by Uncertainty Reduction on Neural Representation and Self-Supervision on Neural Eigenspace](https://mlsanity.com/paper/2102.09896/) (2021)
- [IVD-Net: Intervertebral disc localization and segmentation in MRI with a multi-modal UNet](https://mlsanity.com/paper/1811.08305/) (2018)

## Built on by (most cited first, 1 of 1)

- [LUMOS: Latent Universal Medical Priors for Segmentation](https://mlsanity.com/paper/2603.01115/) (2026)

## Abstract (arXiv)

This study demonstrates a cost-effective approach to semantic segmentation using self-supervised vision transformers (SSVT). By freezing the SSVT backbone and training a lightweight segmentation head, our approach effectively utilizes imperfect labels, thereby improving robustness to label imperfections. Empirical experiments show significant performance improvements over existing methods for various annotation types, including scribble, point-level, and image-level labels. The research highlights the effectiveness of self-supervised vision transformers in dealing with imperfect labels, providing a practical and efficient solution for semantic segmentation while reducing annotation costs. Through extensive experiments, we confirm that our method outperforms baseline models for all types of imperfect labels. Especially under the zero-shot vision-language-model-based label, our model exhibits 11.5%p performance gain compared to the baseline.