# Boosting Low-Data Instance Segmentation by Unsupervised Pre-training with Saliency Prompt

arXiv: https://arxiv.org/abs/2302.01171 · HTML: https://mlsanity.com/paper/2302.01171/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 47 references that are in the map.
- First posted on arXiv 2 Feb 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [Visual Saliency Transformer](https://mlsanity.com/paper/2104.12099/) (2021)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [Salient Object Detection via Integrity Learning](https://mlsanity.com/paper/2101.07663/) (2021)
- [A Survey of Visual Transformers](https://mlsanity.com/paper/2111.06091/) (2021)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [TensorMask: A Foundation for Dense Object Segmentation](https://mlsanity.com/paper/1903.12174/) (2019)
- [Weakly Supervised Object Localization and Detection: A Survey](https://mlsanity.com/paper/2104.07918/) (2021)

## Built on by (most cited first, 11 of 11)

- [GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding](https://mlsanity.com/paper/2311.11863/) (2023)
- [Cross-Platform Video Person ReID: A New Benchmark Dataset and Adaptation Approach](https://mlsanity.com/paper/2408.07500/) (2024)
- [Rapid Salient Object Detection with Difference Convolutional Neural Networks](https://mlsanity.com/paper/2507.01182/) (2025)
- [Unified-modal Salient Object Detection via Adaptive Prompt Learning](https://mlsanity.com/paper/2311.16835/) (2023)
- [AO-DETR: Anti-Overlapping DETR for X-Ray Prohibited Items Detection](https://mlsanity.com/paper/2403.04309/) (2024)
- [Training-Free Dataset Pruning for Instance Segmentation](https://mlsanity.com/paper/2503.00828/) (2025)
- [Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchange](https://mlsanity.com/paper/2404.07504/) (2024)
- [Unsupervised Pre-training with Language-Vision Prompts for Low-Data Instance Segmentation](https://mlsanity.com/paper/2405.13388/) (2024)
- [Masked Collaborative Contrast for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2305.08491/) (2023)
- [Visual Attention Prompted Prediction and Learning](https://mlsanity.com/paper/2310.08420/) (2023)
- [DualGazeNet: A Biologically Inspired Dual-Gaze Query Network for Salient Object Detection](https://mlsanity.com/paper/2511.18865/) (2025)

## Abstract (arXiv)

Recently, inspired by DETR variants, query-based end-to-end instance segmentation (QEIS) methods have outperformed CNN-based models on large-scale datasets. Yet they would lose efficacy when only a small amount of training data is available since it's hard for the crucial queries/kernels to learn localization and shape priors. To this end, this work offers a novel unsupervised pre-training solution for low-data regimes. Inspired by the recent success of the Prompting technique, we introduce a new pre-training method that boosts QEIS models by giving Saliency Prompt for queries/kernels. Our method contains three parts: 1) Saliency Masks Proposal is responsible for generating pseudo masks from unlabeled images based on the saliency mechanism. 2) Prompt-Kernel Matching transfers pseudo masks into prompts and injects the corresponding localization and shape priors to the best-matched kernels. 3) Kernel Supervision is applied to supply supervision at the kernel level for robust learning. From a practical perspective, our pre-training method helps QEIS models achieve a similar convergence speed and comparable performance with CNN-based models in low-data regimes. Experimental results show that our method significantly boosts several QEIS models on three datasets. Code will be made available.